next »

[Shaella]

I always used SourceSafe myself, but if you want to set up a SVN server then hopefully I can figure out how that works.

[Shaella]

If this project is discontinued, let me know... My webcrawler tool has been finished for a while now, but it's all slowing down...

[dastof]

I think that, maybe, lots of people here are student, and this could be a busy time for them?

[Shaella]

I guess maybe you are right

Well, in the meantime I will hold on to MirrorTool and the recoded aoresourcetool then.

[Onni]

Shaella,

Is there a chance you might put the source online or it all available as download?
Even tho the idea was to put it on SVN as Vhab said, but that is a while ago.

[Shaella]

Well, as stupid as it might seem, yesterday I noticed that Vhab sent me a personal message some time back (just after i finished the application).
I never even thought of there being personal messages on this forum so never noticed...

Anyway, I will put it up on SVN as soon as I get the chance.

So I guess we go on as planned again No releasing of the sourcecode atm.

[Shaella]

Initial version of MirrorTool commited to SVN Server.

[axl]

any estimates as to what speed can your tools bring? last i checked i did full org sync + rk1 individual xml's in a little under 1 hour (approximatively 1.8Gb worth of xml's). can it be done faster?

[Shaella]

Actually I haven't tested this too extensively...
Do you only download the xml files or do you parse them and insert them in some relational db as well?

Last I checked with 3 threads running I think it was done with everything (all guilds + guild members + single lookups, downloaded, parsed and inserted/updated into mysql etc) in just under 2 hours or something.
But then I run normal priority on threads and only 3 of them while playing AO...
The limit was 10 hours I think so when it was running that fast I didn't bother to think about the duration anymore hehe.

I can check though if I get the chance one day

[axl]

posted in another thread announcing a xml mirror site. keeps the same structure url wise as people.anarchy-online.com. and it works like this:
when u ask my server for an xml, it will go on people and mirror it. if it's an org xml it will break it down in individual xml's. all files have a maximum age (default seted to 1day + 1hour). when a new / updated individual xml is saved, the data is also interpreted and saved to db. cacheing is done based on "last_updated" field in the xml itself, not the time at the download. this design also allows me to use as many http proxy as i want. there is virtual no limit to how many threads u can download at one time SINCE each xml is an individual program run.
from this point on u can use any threaded downloader/linkchecker to visit all xml's. dont need to parse. dont need to download. personally i use linkchecker(http://linkchecker.sourceforge.net/), a threaded linkchecker (checks for 403/404, anchors etc etc in sites), written in python. it's fast and very reliable.

full source code and db dump for today available at http://xml.ao-divine.org/download.tar.bz2 case someone else wants to check it out.

[Demoder]

I made a mirror of people.anarchy-online.com also.. I'm slightly upset with myself that I missed this thread, as it seems several other people have done this already.
Anyway, here's my information:
I currently provide a guild roster, and individual character lookup of characters covered by the guild roster, adding more soon.

Guild roster:

Code:

url: http://db.flw.nu/porkd/index.php?method=orgid&query=(ORGID)&d=(DIMENSION)
Replace (ORGID) with your organization ID.
Replace (DIMENSION) with 1 for Atlantean, 2 for Rimor, 3 for DNW.

Character XML:

Code:

http://db.flw.nu/porkd/index.php?method=nick&query=(NICKNAME)&d=(DIMENSION)
Replace (NICKNAME) with the nickname
Reaplace (DIMENSION) with 1 for Atlantean, 2 for Rimor, 3 for DNW.

For more detailed information, see the post on the official AO forums