Last Shout - Posted by: Bernie - Sep. 09, 2020, 04:40:16 PM
WTF is up NW?!!!! ;D

Author Topic: Script Extraction Progress.  (Read 2012 times)

0 Members and 1 Guest are viewing this topic.

Offline NightWolve

  • Administrator
  • Distinguished Shogun
  • *****
  • Posts: 865
  • Karma: +132/-0
  • Gender: Male
    • View Profile
    • Ys Utopia.net
Script Extraction Progress.
« on: Sep. 08, 2006, 08:48:44 PM »
Some might have noticed a few days ago I made a separate category for "inactive" projects and I moved this forum in there. Then, in a schizophrenic fashion, I undid it all and restored the forum to what is was originally... The reason was I really wanted to put this project off, but for some reason, I later got the itch to try to do what I originally wanted. That is, just get the damn script to omgfloofy and let her have fun with it for however the hell long she can handle it. So, initially, I tried to figure out how the script files were indexed but I found that is clearly not going to work. But, I always had a 2nd option which was due to the way 99% of the strings are each preceded by a 4 byte value, fairly unique in most cases. I have been building a perl script that takes all the script files, applies many many regular expressions to each file, then writes them all to one single file. Basically, I have to keep identifying all the unique 4 values and add a regular expression to replace them with a unique arbitrary token of my choosing (i.e. <!!>) which will allow me to easily absorb all strings into my database/application translation tool. Once that happens, I can just hand off an editor to omgfloofy and she can work on it to her heart's content.

So far, the script is over 2 MB, over 40,000 strings. I have about 100KB left of S-JIS to write regular expressions for proper preparation, the process I previously described. That's where I'm at now. This last 100KB is being a pain in the ass, but once I finish up, it'll all be in omgfloogy's court. I'll have the exact script figures when I finally manage to finish this phase, hopefully very soon.

This is what the perl script looks like so far if anyone's curious:

Code: [Select]
#!/usr/local/bin/perl
# By: NightWolve
use IO::Handle;
use File::Find;
use Cwd;

my @packfiles  = ('ED6_DT01.bin', 'ED6_DT02xxxx.bin');
my @targetdirs = ('ED6_DT01', 'ED6_DT02');

my $targetdir, $packfile;
my $expectedsize = 4650017;
my $path = cwd;
my $file;
my $size;
my $data;
my $cmd;

my @filter = (

# ['([^\x81-\x9F\xE0-\xFC]{1}\[..)([\x23\x81-\x9F\xE0-\xFC]{1})', '$1ADBC$2' ]
# ,[ '\x54\x06([\x81-\x9F\xE0-\xFC]{1}[\x40-\xFC]{1})', '<>$1' ]
# ,[ '\x01\x01\x23', '<>\x23' ]
# ,[ '\x00\x01([\x81-\x9F\xE0-\xFC]{1}[\x40-\xFC]{1})', '\x00\x00$1' ]
# ,[ '\x00\x54\x07\x02', '\x00\x00><' ]


# All of the \x00\x5B family complete!

 [ '\x00\x5B[\x02\x03\x08-\x1F\x21-\x24\x27-\x2B\x2D\x31\xFE]\x00', '<!!>' ]
,[ '\x00\x5B[\x01-\x08\x0B\x0E-\x10\x12\x33-\x38\x3B]\x01', '<!!>' ]


# \x01\x5B family COMPLETE!

,[ '\x01\x5B[\x02\x03\x08-\x21\x27-\x2B\x31\x32\xFE]\x00', '<!!>' ]
,[ '\x01\x5B[\x01-\x08\x0B\x0E-\x10\x33\x34\x36\x37\x38\x3B\x3C\x22\x2B]\x01', '<!!>' ]


# \x03\x5B FAMILY COMPLETE

,[ '\x03\x5B[\x08-\x10\x14\x1C\x1E]\x00', '<!!>' ]
,[ '\x03\x5B[\x01-\x09\x36\x3F]\x01', '<!!>' ]


# \x04\x5B FAMILY COMPLETE

,[ '\x04\x5B[\x08\x09\x0B\x0C\x1C]\x00', '<!!>' ]
,[ '\x04\x5B[\x01-\x05\x08]\x01', '<!!>' ]


# \x0D\x5B FAMILY COMPLETE

,[ '\x0D\x5B[\x08-\x1A\x1C\x1E\x1F\x20\x21\x24\x31]\x00', '<!!>' ]
,[ '\x0D\x5B[\x01-\x08\x0B\x10\x1C\x36\x3B]\x01', '<!!>' ]

# \x10\x5B FAMILY COMPLETE

,[ '\x10\x5B[\x08-\x11\x15]\x00', '<!!>' ]
,[ '\x10\x5B[\x01\x02\x05\x08\x0B\x3B]\x01', '<!!>' ]


# \x11\x5B FAMILY COMPLETE

,[ '\x11\x5B[\x07-\x0C\x0D\x0F\x10\x18\x1E]\x00', '<!!>' ]
,[ '\x11\x5B[\x01\x02\x07\x08\x10\x3B]\x01', '<!!>' ]


## FAMILY COMPLETE

,[ '\x15\x5B\x22\x00', '<!!>' ]


# ?\x5B\xFE\x00 FAMILY COMPLETE

,[ '.\x5B\xFE\x00', '\x00<!!>' ]


# \x58\x5B FAMILY COMPLETE

,[ '\x58\x5B[\x02\x03\x08-\x24\x27-\x2D\x31\x32\x34\xFE]\x00', '<!!>' ]
,[ '\x58\x5B[\x01-\x08\x0B\x0E-\x10\x33-\x38\x3B\x3F]\x01', '<!!>' ]


# \x58\x5C FAMILY COMPLETE

,[ '\x58\x5C[\x00\x08-\x13\x16\x18\x19\x1B\x1C\x1E\x20]\x00', '<!!>' ]
,[ '\x58\x5C[\x01-\x05\x07\x08\x36\x3F]\x01', '<!!>' ]

# \x22\x5B FAMILY COMPLETE

,[ '\x22\x5B[\x01\x03\x05\x10\x3B]\x01', '<!!>' ]
,[ '\x22\x5B[\x08\x09\x0E\x10]\x00', '<!!>' ]

# \x27\x5B FAMILY COMPLETE

,[ '\x27\x5B[\x08\x0C]\x00', '<!!>' ]
,[ '\x27\x5B[\x01\x08]\x01', '<!!>' ]

# \x08\x5B FAMILY COMPLETE

,[ '\x08\x5B[\x08]\x00([^\x00])', '<!!>$1' ]
,[ '\x08\x5B[\x01]\x01', '<!!>' ]

# \x09\x5B FAMILY COMPLETE

,[ '\x09\x5B[\x08\x2F]\x00([^\x00])', '<!!>$1' ]
,[ '\x09\x5B[\x01\x03\x05\x06\x07\x36]\x01', '<!!>' ]

# \xFF\x5B FAMILY COMPLETE

,[ '\xFF\x5B[\x08\x09\x0A\x0B\x0D\x0F\x11\x13\x14\x18\x27\x31]\x00', '<!!>' ]
,[ '\xFF\x5B[\x01\x03\x05\x06\x08\x0E\x10\x36\x38]\x01', '<!!>' ]


# \x00\x5C

,[ '\x00\x5C[\x08-\x0C\x0E-\x11\x13]\x00', '<!!>' ]
,[ '\x00\x5C[\x0D\x12]\x00([^\x00])', '<!!>$1' ]
,[ '\x00\x5C[\x01\x05\x06\x07\x08\x0E\x36\x3F]\x01', '<!!>' ]


# All this shit down here needs review to be 100% certain!


# 2 BYTE ID apparently

,[ 'XT', '<!!>' ]




#,[ '[\x02\x05\x09\x0B\x0C\x0F\x11\x12\x14\x16\x19\x1A\x1C\x1D\x7C]\x5B\x07\x01', '<!!>' ]

#,[ '.\x5B[\x08\x0C\x11]\x00', '\x00<!!>' ]
#,[ '.\x5B[\x01\x06\x0]\x01', '\x00<!!>' ]



,[ '\x5B\x02\x01\x00', '\x00\x00\x00\x00' ]
,[ '.\x5B\x02\x01', '\x00<!!>' ]

,[ '[\x00\x64\xFF]\x54\x07.', '<!!>' ]
,[ '\xFF\xFF\x54\x06([^\x00])', '<!!>$1' ]
,[ '\xFF\xFF\xFF\x54', '\x00\x00\x00\x00' ]


,['([^\x07]\x00)([\x81-\x9F\xE0-\xFC][\x40-\x7E\x80-\xFC][\x81-\x9F\xE0-\xFC][\x40-\x7E\x80-\xFC][\x81-\x9F\xE0-\xFC][\x40-\x7E\x80-\xFC])', '$1<!!>$2' ]

,['\x00([0-9A-Z ]{5,7}[\x81-\x9F\xE0-\xFC][\x40-\x7E\x80-\xFC][\x81-\x9F\xE0-\xFC][\x40-\x7E\x80-\xFC])', '\x00<!!>$1' ]
);

# Recursive file finder
my $func_ref = sub {
if ( -f )  {
$file = "$path/$File::Find::name";
$size = -s $file;
print "Appending $File::Find::name to $packfile ($size bytes)...\n";
open INP, $file or die "can't open file";
binmode INP, ":raw";
read INP, $data, $size;
close INP;
if ( 1 ) {
for $regpair (@filter) {
$cmd  =  '$data =~ s/' . $regpair->[0]. '/' . $regpair->[1] . '/sg;';
eval ($cmd);
}
$data =~ s/<!!>(\x07\x00|[^\x00])*\x00/\x00/g;
}
#print OUT "!FILE: $File::Find::name\x00";
print OUT $data;
}
};

$i = 0;
for (@packfiles) {
$packfile = $_;
if ( -f $packfile ) {
open OUT, ">$packfile" or die "Can't open $packfile for writing";
binmode OUT, ":raw";
# tell File::Find to locate all files in the directory
find ( {wanted => $func_ref}, @targetdirs[$i]);
close OUT;
}
$i++;
}

$size = -s $packfile;
if ( $size != $expectedsize ) {
print "$packfile is $size while the expected size was $expectedsize\n";
#<STDIN>;
}


You break my record, now I break you, like I break your friend!

Offline MonCapitan

  • Hapless Fledgling
  • *
  • Posts: 24
  • Karma: +28/-0
  • Cute Widdle Moogle
    • View Profile
Re: Script Extraction Progress.
« Reply #1 on: Sep. 10, 2006, 10:31:23 AM »
That is pretty cool.  I just got the game recently and I am at the beginning of the game.  I wish you and omgfloofy much success with this project.

Offline omgfloofy

  • Hapless Fledgling
  • *
  • Posts: 6
  • Karma: +0/-666
  • Gender: Female
    • View Profile
    • swordbreaker.com
Re: Script Extraction Progress.
« Reply #2 on: Sep. 14, 2006, 12:03:43 AM »
This is a rather daunting project. I'm still kinda dizzied over the scale of the script to this game. But then- I also planned to go through and translate it by hand at one point, so... I already know I'm crazy to begin with.

*patpats stacks of random translations from the game* XD;;

I'm terrified to think that when I'm done, that there will be a desire to hit SC. And after playing it, I know that game is going to have a much worse script. *dies*