next »

[helge]

So, I've just figured out some packets from/to the login, character and dimension server. This is just a post to inform people what I've figured out and save them the pain of connecting 4 and 4 bytes together in a packet of 2k length etc I'm sorry in advance for any misspellings and/or uncorrect usage of words :p

They use the same login algorithm as the chat server, and everything is just structured data (like with the chat server), so it's really only a matter of patience to get this implemented into our aocchat-libraries; it shouldn't be hard.

First a little information about how the packets are generally structured. Here's how the header looks like: (I will use C-types to describe size)
[unsigned int dataLength] [unsigned short strLength, char fromProtocol[strLength]] [unsigned int fromId1] [unsigned int fromId2] [unsigned short strLength, char toProtocol[strLength]] [unsigned int toId1] [unsigned int toId2] [unsigned int packetType]
Example: ServerInterface 0 0 ServerAgent 0 0 9
(AFAIK from/toId1-2 can be any arbitrary numbers)

Quite an upgrade in header-size from the chat-server. The protocols i've found so far is UniverseInterface/UniverseAgent, PlayerInterface/PlayerAgent, ServerInterface/ServerAgent, CSPlayerInterface/CSPlayerAgent.
When you send data you usually put the interface first, and the agent after, and reverse when receiving data, but it's not always like that. I havn't really figured out that part yet, I just group them by fromProtocol and so far that's been successful (sometimes it does give you [ServerAgent x x PlayerInterface x x x] though, so reality is probably a bit more complex)

Anyway.. after the header, the arguments comes of course, and these are handled like this:

Legend:
I: unsigned int (4 bytes)
H: unsigned short (2 bytes)
B: unsigned char (1 byte)
S: unsigned short strLen, char[strLen] (2 bytes of length followed by an ASCII string)
1: (special: it's a list of your characters): (1st arg/1000)+1 gives you the count of characters (0-8). Then, for each char: IIISIISIIIIIIII
2: (special: list of all the servers): First arg is size of array, for each server: IIIISHIIIIIII
3: (special: list of your guildies): while there is data: HISBBHHBI

        CSPLAYER: {
            'out': {
                CSPLAYER_HELLO:     'II',
                CSPLAYER_REQUEST_INFO:  'II',
            },
            'in': {
                CSPLAYER_HELLO:     'I',
                CSPLAYER_GUILDINFO:     'IIIIISHSBIIHHHHHHHHHHHHHHHHHHHHHHHHHIIIHIIIIIHISBBHHBIIHH3',
                CSPLAYER_FRIEND: 'HISIIHBBBBB',
            }
        },
        UNIVERSE: {
            'out': {
                UNIVERSE_LOGIN1:    'HSI',
                UNIVERSE_LOGIN2:    'S',
            },
            'in': {
                UNIVERSE_LOGIN1:    'S',
                UNIVERSE_LOGIN2:    'IIISII',
                UNIVERSE_LOGIN_OK:  'IIIIIBHII',
            }
        },
        PLAYER: {
            'out': {
                PLAYER_HELLO:       'II',
                PLAYER_LOGIN:       'IHI',
                PLAYER_UNKNOWN:     'II',
            },
            'in':{
                PLAYER_HELLO:       'I',
                PLAYER_LOGIN:       'IHI',
                PLAYER_CHARLIST:    'I1I',
                PLAYER_SERVERLIST:  '2',
                PLAYER_UNKNOWN2:    'III',
                PLAYER_LOGIN_WAIT:  'I',
            }
        },
        SERVER: {
            'out': {
            },
            'in': {
                SERVER_SERVERINFO:    'IHIHHHIIBIIIIIII',
                SERVER_DIMINFO:    'IHIII',
                SERVER_LOGIN_FAIL:   'I',
            }
        }

with the constants:
UNIVERSE_LOGIN1 = 0
UNIVERSE_LOGIN2 = 1
UNIVERSE_LOGIN_OK = 5

# Player
PLAYER_HELLO = 0
PLAYER_LOGIN = 2
PLAYER_CHARLIST = 4
PLAYER_SERVERLIST = 5

PLAYER_UNKNOWN = 6
PLAYER_UNKNOWN2 = 8
PLAYER_LOGIN_WAIT = 10

# Server
SERVER_SERVERINFO = 2
SERVER_DIMINFO = 3
SERVER_LOGIN_FAIL = 10

# CSPlayer
CSPLAYER_HELLO = 0
CSPLAYER_REQUEST_INFO  = 25
CSPLAYER_FRIEND = 38
CSPLAYER_GUILDINFO = 44

Note: The structure shown above is probably mostly wrong; I've just tried to get to the dimension server and getting the Guildinfo-packet, not been thinking much of the other packets so far. The above structures should leave you with no packets with data unparsed though.

From authentication server -> dimension server:
Note: When I say for instance "Send a CSPLAYER_HELLO" you must start the packet with a CSPlayerInterface/Agent header. An example: "send UNIVERSE_LOGIN2 with the key as argument 1" has the header [UniverseInterface 0 0 UniverseAgent 0 0 2] and [key] as argument.
1. connect to 213.244.186.138:7000, send it UNIVERSE_LOGIN1 with arguments [0, username+":2", 1] (if username is helge, 'helge:2')
2. receive key hash in argument 1 (arg[0]) from the next packet
3. just call your favourite generate_ao_login_key function in the same manner as before, and send the key as argument 1 with packetid UNIVERSE_LOGIN2
4. receive 2 packets. The last one will contain ip/port to the characterServer in argument 4, and an important session ID in argument 5.
--
5. Use this information to connect to the characterServer, and send a PLAYER_HELLO with sessionid as the 2. argument
6. Receiving packet should be a LOGIN_OK (or something like that)
7. Here I send a PLAYER_UNKNOWN with the arguments [10002, 1002715599] - Don't ask me why (I think both these are gotten from the authentication server, but I also think they are static, so they should work.. it's like that with 50% of all the arguments)
8. Next packet received contains your characterlist.
9. Next packet received contains the serverlist.
10. Use the data collected from the charlist to send a PLAYER_LOGIN with arguments [charId, 0, 1002715599
11. Receive a packet and do inet_ntoa on argument 1 (long to ip), and your port is in argument 2. That's right, no more static XML files with ip addresses and bad spelling!
12. Go connect to your dimension server and send a CSPLAYER_HELLO with arguments [charid, sessionid] (sessionid is the same as the one you got from authentication server)
13. Receive a CSPLAYER_HELLO back that tells you you're in
14. Send a CSPLAYER_REQUEST_INFO with the arguments [50000, charId] where charId is your own character's id.
15. Receive a huge packet (CSPLAYER_GUILDINFO) containg guild info like name, motd, and a list of all the players in your guild.
16. Just start poll packets infinitly (you will get lots of CSPLAYER_FRIEND you can use to find out if a player connects/disconnects/levels up and even zone or get promoted)

[noer]

Good job, havn't been home lately so I havn't had time to look into that.

[Trosky]

I think, Funcom could react quite allergic, if we start to connect to the dimension servers :/

[Mafoo]

This is fantastic news! will probably keep me busy on the weekend turning it into a lib for perl to compliment the AOC::Chat.pm

Good work on the investigation!

[POD13]

This is an interesting post, helge.

From your description it sounds like you managed to duplicate the group-guild-bug that happens when people crash and log back in to find they have no group or guild list anymore. If you remain out of the game long enough that you get the "so-and-so logged out" message, you would probably get the guild info on your next login. I hope you find a way to log out of the dimension server. Maybe that will lead to a fix for this very annoying bug.

[helge]

This is fantastic news! will probably keep me busy on the weekend turning it into a lib for perl to compliment the AOC::Chat.pm

Good work on the investigation!

Good to see someone else working on this:-)

I will start making a lib in Python from scratch soon(both chat/dimension..). I just made one in c++, but I'm not very happy with it (everything works, but I'm just not comfortable with the language; debugging the protocol while learning a language isn't easy), but if anyone wants the code for their own debugging, just ask and I'll upload it somewhere.

Good luck and remember to post your findings!

[helge]

This is an interesting post, helge.

From your description it sounds like you managed to duplicate the group-guild-bug that happens when people crash and log back in to find they have no group or guild list anymore. If you remain out of the game long enough that you get the "so-and-so logged out" message, you would probably get the guild info on your next login. I hope you find a way to log out of the dimension server. Maybe that will lead to a fix for this very annoying bug.

While I havn't been researching this much since my first post, I don't think it's related to that. The packet will always be sent if you connect with AoC.exe (well, I havn't experienced the bug you were describing, but still..), but it's never sent with your own sockets. Also, the guild information packet is sent every time you zone ingame, but I can't see any packets requesting it (on the dimension server).. My theory atm is that this is requested through the game server.
I have no idea why I got the packet once, though. My current theory on THAT is that I was hallucinating

PS. logging into the game and logging out again doesn't fix it. I can't even get the packet on newly created chars.

[Mafoo]

ok run into some inconsistencies (what funcom inconsistent? no!) (Looking at the packet captures
During the communication with the UNIVERSE server for packet type 0 outbound the S is encoded as
[An unsigned long (32-bit) in "network" (big-endian) order][A string with arbitrary binary data, will be null padded]
being
[Length of String][String]

however during outbound communication for packet type 1 the S is encoded as
[An unsigned short (16-bit) in "network" (big-endian) order][A string with arbitrary binary data, will be null padded]
being
[Length of String][String]

If i either match this behaviour or leave it as a 32bit number i cannot seem to logon. get the return UNIVERSE_LOGON1 (type 1) with some funny numbers with no preceding UNIVERSE_LOGON_OK (type 5)

Code:

0000005C  00 00 00 4c 00 0d 55 6e  69 76 65 72 73 65 41 67 ...L..Un iverseAg
0000006C  65 6e 74 00 00 00 01 00  00 00 00 00 11 55 6e 69 ent..... .....Uni
0000007C  76 65 72 73 65 49 6e 74  65 72 66 61 63 65 00 11 verseInt erface..
0000008C  b5 81 00 00 00 00 00 00  00 01 ff ff ff ff 00 00 ........ ........
0000009C  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 01 ........ ........

instead of the expected (from packet capture)

Code:

000000B3  00 00 00 6c 00 0d 55 6e  69 76 65 72 73 65 41 67 ...l..Un iverseAg
000000C3  65 6e 74 00 00 00 06 00  00 00 00 00 11 55 6e 69 ent..... .....Uni
000000D3  76 65 72 73 65 49 6e 74  65 72 66 61 63 65 3b bb verseInt erface;.
000000E3  4c 87 00 00 00 00 00 00  00 01 00 00 00 01 00 00 L....... ........
000000F3  27 12 3b bb 4c 87 00 20  70 72 6f 64 64 6d 30 31 '.;.L..  proddm01
00000103  2e 61 6d 73 2e 61 67 65  6f 66 63 6f 6e 61 6e 2e .ams.age ofconan.
00000113  63 6f 6d 3a 37 30 33 37  48 c4 ea b9 00 00 00 00 com:7037 H.......

[helge]

ok run into some inconsistencies (what funcom inconsistent? no!) (Looking at the packet captures
During the communication with the UNIVERSE server for packet type 0 outbound the S is encoded as
[An unsigned long (32-bit) in "network" (big-endian) order][A string with arbitrary binary data, will be null padded]
being
[Length of String][String]

however during outbound communication for packet type 1 the S is encoded as
[An unsigned short (16-bit) in "network" (big-endian) order][A string with arbitrary binary data, will be null padded]
being
[Length of String][String]

If i either match this behaviour or leave it as a 32bit number i cannot seem to logon. get the return UNIVERSE_LOGON1 (type 1) with some funny numbers with no preceding UNIVERSE_LOGON_OK (type 5)

I defined outgoing packet 0 as HSI, which fixes that:
"0x00 0x00 0x00 0x0a helgefmi:2 .." becomes [0, "helgefmi:2", ..].

I find it weird if funcom is meaning that to be SI (with string length defined as a long), since every other strings has length defined as a short. Besides, who want a username with more than 65535 chars anyway. I'm guessing the first two bytes are an argument of itself.

Going for vacation now, cya in a week (unless I find some open WLAN's:-))

[Mafoo]

Thanks, weirdly making that adjustment to the code has made my logon process actually pass the Universe logon!
during testing i identified another packet. It looks like a error packet as i got it back when i encoded my Universe_logon1 wrong, comes in as a type 2 and is defined below, the single parameter (i think) comes in as a int of a value 12

for those interested from a different language point of view, this is how i am storing the packet definitions for the Perl version.
I am storing the meta information to save me having to figure which to send each time, the packet def takes care of it

Code: (Perl)

our %PACKETS = (
                 UNIVERSE => {
                               'OUT' => {
                                          0 => {
                                                 Name            => 'UNIVERSE_LOGIN1',
                                                 Attribute1      => 'UniverseInterface',
                                                 Attribute1Meta1 => '1002130567',
                                                 Attribute1Meta2 => '0',
                                                 Attribute2      => 'UniverseAgent',
                                                 Attribute2Meta1 => '0',
                                                 Attribute2Meta2 => '0',
                                                 Params          => 'HSI',
                                          },
                                          1 => {
                                                 Name            => 'UNIVERSE_LOGIN2',
                                                 Attribute1      => 'UniverseInterface',
                                                 Attribute1Meta1 => '1002130567',
                                                 Attribute1Meta2 => '0',
                                                 Attribute2      => 'UniverseAgent',
                                                 Attribute2Meta1 => '0',
                                                 Attribute2Meta2 => '0',
                                                 Params          => 'S',
                                          },
                               },
                               'IN' => {
                                         0 => {
                                                Name            => 'UNIVERSE_LOGIN1',
                                                Attribute1      => 'UniverseAgent',
                                                Attribute1Meta1 => '1',
                                                Attribute1Meta2 => '0',
                                                Attribute2      => 'UniverseInterface',
                                                Attribute2Meta1 => '0',
                                                Attribute2Meta2 => '0',
                                                Params          => 'S',
                                         },
                                         1 => {
                                                Name            => 'UNIVERSE_LOGIN2',
                                                Attribute1      => 'UniverseAgent',
                                                Attribute1Meta1 => '6',
                                                Attribute1Meta2 => '0',
                                                Attribute2      => 'UniverseInterface',
                                                Attribute2Meta1 => '0',
                                                Attribute2Meta2 => '0',
                                                Params          => 'IIISII',
                                         },
                                         2 => {
                                                Name            => 'Malformed Packet',
                                                Attribute1      => 'UniverseAgent',
                                                Attribute1Meta1 => '0',
                                                Attribute1Meta2 => '0',
                                                Attribute2      => 'UniverseInterface',
                                                Attribute2Meta1 => '0',
                                                Attribute2Meta2 => '0',
                                                Params          => 'I',
                                         },
                                         5 => {
                                                Name            => 'UNIVERSE_OK',
                                                Attribute1      => 'UniverseAgent',
                                                Attribute1Meta1 => '0',
                                                Attribute1Meta2 => '0',
                                                Attribute2      => 'UniverseInterface',
                                                Attribute2Meta1 => '0',
                                                Attribute2Meta2 => '0',
                                                Params          => 'IIIIICHII',
                                         },
                               }
                 },

[Mafoo]

more on this, i am now on the character server stage I disagree with a couple of your decoding

for the character list it is (in pack form)
NN(NNNn/a*NNn/a*NNNNNNNN)8

being the following attributes
N: Your SEED (used as your transaction ID for Attribute1Meta1)
N: Don't know
for each repeating
N: character number
N: Your SEED
N: character number
S: Character name
N: Server index
N: normally 0, unknown
S: a date
N: normally 0, unknown
N: unknown
N: level
N: class
N: normally 0, unknown
N: large number seems to always be the same
N: normally 3
N: normally 1

for the server it is (in pack form)

NXXXXN/(NNNNn/a*nNNNNNNN)*
N: number of servers
Repeatables
N: ServerID
N: 2
N: 8
N: 10000
S: Server Name
n: 0
N: 0
N: 0
N: 0
N: 0
N: 0
N: 256
N: 0

[helge]

Thanks, weirdly making that adjustment to the code has made my logon process actually pass the Universe logon!
during testing i identified another packet. It looks like a error packet as i got it back when i encoded my Universe_logon1 wrong, comes in as a type 2 and is defined below, the single parameter (i think) comes in as a int of a value 12

for those interested from a different language point of view, this is how i am storing the packet definitions for the Perl version.
I am storing the meta information to save me having to figure which to send each time, the packet def takes care of it

Code: (Perl)

our %PACKETS = (
                 UNIVERSE => {
                               'OUT' => {
                                          0 => {
                                                 Name            => 'UNIVERSE_LOGIN1',
                                                 Attribute1      => 'UniverseInterface',
                                                 Attribute1Meta1 => '1002130567',
                                                 Attribute1Meta2 => '0',
                                                 Attribute2      => 'UniverseAgent',
                                                 Attribute2Meta1 => '0',
                                                 Attribute2Meta2 => '0',
                                                 Params          => 'HSI',
                                          },
 

To be honest I don't see the point in saving the numbers statically like that. You can basically just send "protocol 0 0 protocol 0 0", and it will always work (i think) if you don't care about doing it right, but if you do care you should instead make it more dynamic (explained below).

I think maybe this is how the system works:
Every packet is sent as:
[from protocol/interface] id 0 [to protocol/interface] id 0.
UniverseInterface 1 0 UniverseAgent 0 0 means "UniverseInterface with id 1 is calling UniverseAgent with id 0."
So when the server tells you that your CSPlayerInterface has the ID 33652294, you should instead store that ID and use it (maybe use a map and pull the appropriate id with for example ids['ServerInterface'], and use 0 if it's not set yet..). This is because the ID's will probably change with AoC updates, which character you login with and which dimension server you use etc.

[helge]

more on this, i am now on the character server stage I disagree with a couple of your decoding

for the character list it is (in pack form)
NN(NNNn/a*NNn/a*NNNNNNNN)8

being the following attributes
N: Your SEED (used as your transaction ID for Attribute1Meta1)
N: Don't know
for each repeating
N: character number
N: Your SEED
N: character number
S: Character name
N: Server index
N: normally 0, unknown
S: a date
N: normally 0, unknown
N: unknown
N: level
N: class
N: normally 0, unknown
N: large number seems to always be the same
N: normally 3
N: normally 1

for the server it is (in pack form)

NXXXXN/(NNNNn/a*nNNNNNNN)*
N: number of servers
Repeatables
N: ServerID
N: 2
N: 8
N: 10000
S: Server Name
n: 0
N: 0
N: 0
N: 0
N: 0
N: 0
N: 256
N: 0

I actually have the exact same defined character/serverlist structure as you use (I think.. dunno what you mean by 'n/a*' but assuming it is an S :-)) , but I see now that I forgot some I's on the servers, and mixed up H with S on the characters; will edit that now.
I might misunderstand your post, but the characters is not repated 8 times. The number before the repeatables (arg 2) actually tells you how many characters to loop through.. it gives a number like 7063 for 6, 8072 for 7 etc.. I just do count = (n/1000)-1 where (7063/1000)-1 == 6. something like n%100/10 should work too

[helge]

Writing a little update as I've discovered a couple of new things..
- SERVER incomming packet with type 2 (SERVER_SERVERINFO, previously called SERVER_UNKNOWN I think) has this structure: IHIHHHIIBIIIIIII
- In SERVER incomming packet with type 2 (same as above) you are given the ip/port to some servers in arg1-4. Still not found the Chatserver ip/port, but I'm sure they're hidden in there somewhere. Too lazy to check now
- In PLAYER incomming packet with type 4 (Charlist) arg9 is the playfield
- If you login to the dimension server while already ingame on the client, the client will be disconnected, and you will get an error on the bot. You then have to wait ~1min before reconnecting it.
- If you login to the dimension server with the bot first, you can connect/disconnect as you like with AgeOfConan.exe.
- After logging in to the dimension server with your own sockets, you must wait ~1min before you can connect with AgeOfConan.exe
- You can request the CSPLAYER_GUILDINFO packet (giving you list of guild members with class/lvl/online status etc) simply by sending a CSPLAYER packet with type 25 with the arguments [50000, yourCharId]!!
- I have deciphered the GUILDINFO packet, check details in the (edited) first post
- CSPLAYER incomming packet with type 38 (CSPLAYER_FRIEND, the one who can tell us lvlups, quit/joins, promotions, zones, etc) is probably structured like this:

Legend:
H: 2 bytes integer
I: 4 bytes integer
B: 1 byte integer
S: 2 bytes integer of length followed by a string of that length

Structure:
HISBBHHBII

Example arguments:
[771, 33654182, 'Aocbot', 2, 1, 0, 4022, 6, 29, 1215863940]

arg1: always 771 for me, might be an ID connected to your guild? Probably not, it's only 2 bytes. Post here if anyone has a different number
arg2: charId
arg3: charName
arg4: The rank in the guild (0-4)
arg5: Status (0=offline, 1=online, 2/3 is LFG/AFK, don't remember which is which)
arg6: always 0
arg7: Playfield ID
arg8: level
arg9: class
arg10: unknown. This can suddenly change without anything happening ingame (i think). Maybe this shouldn't be an I, but I can't make sense of it either way.

You will get this packet if the player is pro/demoted, zones somewhere, levels up, disconnects or connects

Edit: I figured out how to request the guildinfo packet and made some changes. Read it again :p

[bheam]

Hi,

Having gone through this, there are some inconsistencies.

The character struct in the guild packet is: HISBBHHBII (mind the extra I at the end), same as the FRIEND struct.

The guild packet is not IIIIISHSBIIHHHHHHHHHHHHHHHHHHHHHHHHHIIIHIIIIIHISBBHHBIIHH3 - this is way too much data (on my side anyway).
For me, it is 'IIIIISHSBIIHHHHHHHHHHHHHHHH[HISBBHHBII]HH3' (I don't understand most of the packet so i just removed enough data for it to work.)

Similarly, character list packet is not "I1I" but "II1".

The last arg in the friend/char struct is a timestamp, probably when the game last saw that person online; so it might be up-to-date every time FRIEND is sent if that person is already online.

As for chat server address; Wild guess: it's same as dimension server, and port is 7000 + server id (Works for fury, confirm for your own server please)

Question though; i see another thread talking about item stats being server side - when i log traffic i can't see anything indicating downloading anything when i click an item link in chat + it's superfast to open a stats window, can't possibly be getting the data realtime. Client can't be downloading every item page from chat before i even click them could it? How would that be for waste of traffic... But well, I just wish the item stats were stored locally, since it's obviously transferred through game server if anything :p

Michael