Tech – Page 10 – Ryan Schulze

How to increase Fraps performance with a ramdisk

November 10, 2011September 27, 2014Ryan2 Commentsbattlefield 3, fraps, IO, powershell, ramdisk, Windows

I recently started playing Battlefield 3 and remembered that I have a Fraps license so I installed it and started recording some stuff. Unsurprisingly the performance made a big dip when I recorded. A glance at my PC told me the harddrive was at fault, probably bringing the whole system down due to IO.

Since my PC has more than enough RAM I decided to set up a 5Gb Ramdisk to see if that helped. It did, when writing the video files to the ramdisk I hardly had any performance hit. Unfortunately 5GB isn’t going to last long while recording 1920×1080 @ 40FPS (a few minutes footage at most).

Here is my little cmd file to create a 5GB ramdisk as drive J: and format it for usage:

imdisk -d -m J:
imdisk -a -t vm -s 5G -o rw -m J:
format J: /A:64k /V:Ramdisk /FS:NTFS /Q

imdisk -d -m J:

imdisk -a -t vm -s 5G -o rw -m J:

format J: /A:64k /V:Ramdisk /FS:NTFS /Q

So my next thought was to see if I could write a script to move files off the ramdisk when they were done being written to by Fraps. This obviously was going to cause IO load … the reason we were having performance issues in the first place, so I was skeptical about if this was going to help any. Especially since I also had to move the files away quick enough so that the drive wouldn’t fill up completely with the next file Fraps was writing. I wrote a little powershell script for this (yeah, a *nix Sysadmin writing scripts in powershell …)

Here is my little powershell script to copy the finished files from my ramdisk to a normal HDD (please excuse possible ugliness, I’m a powershell noob):

$source = "j:\*.avi"
$target_dir = "f:\fraps\movies\"

while ($true) {
    $files = gci $source
    foreach ($file in $files){ 
        $srcfile = echo $file.fullname
        $dstfile = echo $file.basename
        if ($srcfile) {
            try {
                move-item "$srcfile" "$target_dir$dstfile.avi" -ea stop
                "$srcfile Moved"
            }
            catch {
                "$srcfile in use, cannot be copied right now"
            }
        }
    }
    sleep 5
}

$source = "j:\*.avi"

$target_dir = "f:\fraps\movies\"

while ($true) {

$files = gci $source

foreach ($file in $files){

$srcfile = echo $file.fullname

$dstfile = echo $file.basename

if ($srcfile) {

try {

move-item "$srcfile" "$target_dir$dstfile.avi" -ea stop

"$srcfile Moved"

}

catch {

"$srcfile in use, cannot be copied right now"

}

sleep 5

}

The last little problem I noticed is that the 5GB ramdrive wasn’t big enough (Fraps seems to create some dummy files and fills them up). Forcing Fraps to make smaller files by toggeling the recording fixed that though -> pressing F9 twice fast will drop a few frames though. I used my Logitech G13 for that, just had a key mapped to press F9 quickly every 60 seconds. The shortest gap I could get working reliably is 50ms.

This all probably sounds awfully complicated, but it works and solves my problem. Fraps is great software, but it would be immensly helpful if you could set the file size in the settings (instead of it defaulting to 4GB). Or, even better, if Fraps could rework their IO system to work more efficiently.

So to sum everything up:
– create ramdrive
– start script that copies files from the ramdrive to a normal HDD
– set fraps to store videos on the ramdrive
– start game, press F9 to start recording and then press the G13 key to toggle the F9 periodically

How to build an efficient GeoIP SQL table

August 22, 2011April 10, 2012Ryan1 Commentbash, GeoIP, MySQL

This here is a very handy little script I threw together to generate a geoip.sql table for quickly determining which country a IP is from. I already hear you saying “Just convert the IP to an INT and use BETWEEN, how hard can it be”. And you are right, that works. And it may even be your easiest solution, but it just isn’t fast. And if you are planning on hammering the table with thousands of queries you are going to end up looking for something fast.

A while back I found a very interesting posting at www.jcole.us that described how to use Spacial Indexes together with MySQL’s GIS to speed up the queries. The posting has been online for a while and both it and the replies are worth reading.

All I did was make a small bash script to download the current “lite” version of GeoIP CSV file from maxmind.com, use the information from the posting to throw/transform it into a local database table and dump out a .sql file that can be easily imported into any other database. The script isn’t failproof though, it expects your user to be able to use mysql and have permission to create databases/tables and “load data local infile”.

generate_geoip_sql.sh

#!/bin/bash
#===============================================================================
#
#          FILE: generate_geoip_sql.sh
#
#         USAGE: ./generate_geoip_sql.sh
#
#   DESCRIPTION: Generates a optimized sql dump with geoipdata from maxmind.com
#
#         USAGE: import sql into a database
#
#  REQUIREMENTS: mysql database rights to create databases and LOAD DATA LOCAL INFILE
#
#       EXAMPLE:
#                 SELECT country_code,country_name
#                 FROM   geo_ip
#                 WHERE  MBRCONTAINS(ip_poly, POINTFROMWKB(POINT(INET_ATON('1.2.3.4'), 0)))
#
#                 SELECT country_code,country_name
#                 FROM   geo_ip
#                 WHERE  INET_ATON('1.2.3.4') 
#                 BETWEEN ip_from AND ip_to;
#
#        AUTHOR:  Ryan Schulze (rs), ryan@dopefish.de
#
#===============================================================================

set -o nounset                              # Treat unset variables as an error

CSVFile="GeoIPCountryWhois.csv"

if [[ -e ${CSVFile} ]]
  then
  echo -en "\n${CSVFile} already exists, download a newer version? [Y/n]: "
  read answer
  if [[ "$(echo $answer|tr "N" "n")" != "n" ]]
  then
    rm ${CSVFile}
  fi
fi

if [[ ! -e ${CSVFile} ]]
then
  tmpfile=$(mktemp)
  wget -O ${tmpfile} http://geolite.maxmind.com/download/geoip/database/GeoIPCountryCSV.zip
  unzip ${tmpfile} &amp;&amp; rm ${tmpfile}
fi

echo -e "\ngenerating geoip.sql"

mysql -e "create database if not exists temp"
mysql -D temp -e "drop table if exists geo_ip"

echo '
CREATE TABLE geo_ip
(
  id           INT UNSIGNED  NOT NULL auto_increment,
  ip_poly      POLYGON       NOT NULL,
  ip_from      INT UNSIGNED  NOT NULL,
  ip_to        INT UNSIGNED  NOT NULL,
  country_code CHAR(2)       NOT NULL,
  country_name CHAR(50)      NOT NULL,
  PRIMARY KEY (id),
  SPATIAL INDEX (ip_poly)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
LOAD DATA LOCAL INFILE "GeoIPCountryWhois.csv"
INTO TABLE geo_ip
FIELDS
  TERMINATED BY ","
  ENCLOSED BY "\""
LINES
  TERMINATED BY "\n"
(
  @ip_from_string, @ip_to_string,
  @ip_from, @ip_to,
  @country_code, @country_string
)
SET
  id      := NULL,
  ip_from := @ip_from,
  ip_to   := @ip_to,
  ip_poly := GEOMFROMWKB(POLYGON(LINESTRING(
    /* clockwise, 4 points and back to 0 */
    POINT(@ip_from, -1), /* 0, top left */
    POINT(@ip_to,   -1), /* 1, top right */
    POINT(@ip_to,    1), /* 2, bottom right */
    POINT(@ip_from,  1), /* 3, bottom left */
    POINT(@ip_from, -1)  /* 0, back to start */
  ))),
  country_code := @country_code,
  country_name := @country_string
;
' | mysql --local-infile=1 -D temp

mysqldump --opt temp geo_ip &gt; geoip.sql

echo -e "\ndone generating geoip.sql"

#!/bin/bash

#===============================================================================

# FILE: generate_geoip_sql.sh

# USAGE: ./generate_geoip_sql.sh

# DESCRIPTION: Generates a optimized sql dump with geoipdata from maxmind.com

# USAGE: import sql into a database

# REQUIREMENTS: mysql database rights to create databases and LOAD DATA LOCAL INFILE

# EXAMPLE:

# SELECT country_code,country_name

# FROM geo_ip

# WHERE MBRCONTAINS(ip_poly, POINTFROMWKB(POINT(INET_ATON('1.2.3.4'), 0)))

# SELECT country_code,country_name

# FROM geo_ip

# WHERE INET_ATON('1.2.3.4')

# BETWEEN ip_from AND ip_to;

# AUTHOR: Ryan Schulze (rs), ryan@dopefish.de

#===============================================================================

set -o nounset # Treat unset variables as an error

CSVFile="GeoIPCountryWhois.csv"

if [[ -e ${CSVFile} ]]

then

echo -en "\n${CSVFile} already exists, download a newer version? [Y/n]: "

read answer

if [[ "$(echo $answer|tr "N" "n")" != "n" ]]

then

rm ${CSVFile}

if [[ ! -e ${CSVFile} ]]

then

tmpfile=$(mktemp)

wget -O ${tmpfile} http://geolite.maxmind.com/download/geoip/database/GeoIPCountryCSV.zip

unzip ${tmpfile} && rm ${tmpfile}

echo -e "\ngenerating geoip.sql"

mysql -e "create database if not exists temp"

mysql -D temp -e "drop table if exists geo_ip"

echo '

CREATE TABLE geo_ip

(

id INT UNSIGNED NOT NULL auto_increment,

ip_poly POLYGON NOT NULL,

ip_from INT UNSIGNED NOT NULL,

ip_to INT UNSIGNED NOT NULL,

country_code CHAR(2) NOT NULL,

country_name CHAR(50) NOT NULL,

PRIMARY KEY (id),

SPATIAL INDEX (ip_poly)

) ENGINE=MyISAM DEFAULT CHARSET=latin1;

LOAD DATA LOCAL INFILE "GeoIPCountryWhois.csv"

INTO TABLE geo_ip

FIELDS

TERMINATED BY ","

ENCLOSED BY "\""

LINES

TERMINATED BY "\n"

(

@ip_from_string, @ip_to_string,

@ip_from, @ip_to,

@country_code, @country_string

)

SET

id := NULL,

ip_from := @ip_from,

ip_to := @ip_to,

ip_poly := GEOMFROMWKB(POLYGON(LINESTRING(

/* clockwise, 4 points and back to 0 */

POINT(@ip_from, -1), /* 0, top left */

POINT(@ip_to, -1), /* 1, top right */

POINT(@ip_to, 1), /* 2, bottom right */

POINT(@ip_from, 1), /* 3, bottom left */

POINT(@ip_from, -1) /* 0, back to start */

))),

country_code := @country_code,

country_name := @country_string

;

' | mysql --local-infile=1 -D temp

mysqldump --opt temp geo_ip > geoip.sql

echo -e "\ndone generating geoip.sql"

win7 system restore … aaarrrrgggh

July 29, 2011Ryanjunctions, system restore, user directory, win7

Warning: this is mainly me just ranting about Microsoft stupidity and a warning for anyone using junctions.

About a month or two ago I built myself a new PC (old one was dying) and installed Win7 on it. Since I was previously still on WinXP there were a few things to get used to, but all in all I enjoyed the update. There were of course some strange Windows pitfalls involved, moving your users directory to a different drive really isn’t straightforward, but works if you use junctions to link the directories.

Since Virtualbox drops the guests in a subfolder of the users directory I decided to move my users directory to one of the larger mechanical disks in my system, to keep the windows drive (a SSD disk) with plenty of space.

Now fast forward to yesterday, where some software I installed decided to clear out most of my Start->All Programs folder, leaving me with lots of installed softawre, and no Start Menu links. After trying some stuff out I finally caved in and used System Restore to get my Start Menu working again. System Restore actually worked fine, I got my Start Menu back … unfortunately it also corrupted my profile, and while googling to find out how to fix it I found a whole lot of people with the same problem.

Junction + User directory + System Restore = corrupt user, non repairable. Hello Microsoft, this is all original system software, no 3rd party stuff, breaking my user is totally uncool.

So after trying all kinds of crazy tips I found I gave up and followed the official instructions (a.k.a. You are fucked, haha bye bye user registry). And right now I am copying stuff from my old profile to my new one and setting up all the software that had stored stuff in the registry. This time my Profile stays on C: without any junctions, it’s easier to repair Virtualbox if it breaks over the images being in a non-standard path than worrying about windows corrupting my whole profile if I touch System Restore.

How to install ruby 1.9.2 on Ubuntu 10.04

July 7, 2011May 13, 2013Ryan3 Commentsbash, Debian, LTS, ruby, Ubuuntu

The current LTS version of Ubuntu is 10.04 and the most current version of ruby it ships with is 1.9.1. Unfurtunately 1.9.1 wasn’t that great of a release and anyone using the 1.9 branch really should use the stable 1.9.2.

After doing a bit of researching I found some information on how the best approach to get ruby installed is. Downloading the source, compiling it and registering the installed version with the package manager.

The following little bash script takes care of installing ruby 1.9.2 on a ubuntu or debian based system (or any other version if you change the $Version variable in the script). The script just consolidates information found online and wraps it up into a nice bashscript

#!/bin/bash
#===============================================================================
#
#          FILE:  install_ruby_1.9.sh
# 
#         USAGE:  ./install_ruby_1.9.sh 
# 
#        AUTHOR: Ryan Schulze (rs), ryan@dopefish.de
#       CREATED: 07/07/2011 11:59:37 AM CDT
#===============================================================================

Version="1.9.2-p180"
GZFile="ruby-${Version}.tar.gz"
Download="http://ftp.ruby-lang.org/pub/ruby/1.9/${GZFile}"

if [[ "$(id -u)" != "0" ]]
then
	echo "You need root permission to execute this script"
	exit
fi

apt-get -q update
apt-get -qy upgrade
apt-get install -qy build-essential wget zlib1g-dev libssl-dev libffi-dev autoconf

cd /usr/local/src/
test -e ${GZFile} || wget ${Download}
tar -xzf ${GZFile}
cd ruby-${Version}

autoconf
./configure --with-ruby-version=${Version} --prefix=/usr --program-suffix=${Version} 
make
make install

mkdir -p /usr/lib/ruby/gems/${Version}/bin

update-alternatives \
	--install /usr/bin/ruby ruby /usr/bin/ruby${Version} $(echo ${Version//./}|cut -d- -f1) \
	--slave   /usr/share/man/man1/ruby.1.gz ruby.1.gz /usr/share/man/man1/ruby${Version}.1 \
	--slave   /usr/lib/ruby/gems/bin        gem-bin   /usr/lib/ruby/gems/${Version}/bin \
	--slave   /usr/bin/erb  erb  /usr/bin/erb${Version} \
	--slave   /usr/bin/irb  irb  /usr/bin/irb${Version} \
	--slave   /usr/bin/rdoc rdoc /usr/bin/rdoc${Version} \
	--slave   /usr/bin/ri   ri   /usr/bin/ri${Version} \
	--slave   /usr/bin/gem  gem  /usr/bin/gem${Version} \

update-alternatives --config ruby
update-alternatives --display gem &gt;/dev/null 2&gt;&amp;1 &amp;&amp; update-alternatives --remove-all gem

echo "[+] Done installing"
ruby -v

#!/bin/bash

#===============================================================================

# FILE: install_ruby_1.9.sh

# USAGE: ./install_ruby_1.9.sh

# AUTHOR: Ryan Schulze (rs), ryan@dopefish.de

# CREATED: 07/07/2011 11:59:37 AM CDT

#===============================================================================

Version="1.9.2-p180"

GZFile="ruby-${Version}.tar.gz"

Download="http://ftp.ruby-lang.org/pub/ruby/1.9/${GZFile}"

if [[ "$(id -u)" != "0" ]]

then

echo "You need root permission to execute this script"

exit

apt-get -q update

apt-get -qy upgrade

apt-get install -qy build-essential wget zlib1g-dev libssl-dev libffi-dev autoconf

cd /usr/local/src/

test -e ${GZFile} || wget ${Download}

tar -xzf ${GZFile}

cd ruby-${Version}

autoconf

./configure --with-ruby-version=${Version} --prefix=/usr --program-suffix=${Version}

make

make install

mkdir -p /usr/lib/ruby/gems/${Version}/bin

update-alternatives \

--install /usr/bin/ruby ruby /usr/bin/ruby${Version} $(echo ${Version//./}|cut -d- -f1) \

--slave /usr/share/man/man1/ruby.1.gz ruby.1.gz /usr/share/man/man1/ruby${Version}.1 \

--slave /usr/lib/ruby/gems/bin gem-bin /usr/lib/ruby/gems/${Version}/bin \

--slave /usr/bin/erb erb /usr/bin/erb${Version} \

--slave /usr/bin/irb irb /usr/bin/irb${Version} \

--slave /usr/bin/rdoc rdoc /usr/bin/rdoc${Version} \

--slave /usr/bin/ri ri /usr/bin/ri${Version} \

--slave /usr/bin/gem gem /usr/bin/gem${Version} \

update-alternatives --config ruby

update-alternatives --display gem >/dev/null 2>&1 && update-alternatives --remove-all gem

echo "[+] Done installing"

ruby -v

Using iFolder as an open source Dropbox alternative

June 25, 2011May 13, 2013Ryan8 CommentsDropbox, files, iFolder, Internet Stuff, sync

I’ve had a few posting here in my blog over the years about trying to keep data synced over various computers. Nowadays it has gotten easier since it seems everyone is chruning out their own online storage. Classics like Dropbox or SugarSync, and then we have the big companies Google, Microsoft and Ubuntu trying their luck. Basically they all provide you with 2-5Gb free storage, and more space and features for more money. But they all have one downside, whether encrypted or not, you are leaving your own files to the mercy of an external company. There is not really any voodoo in the whole system, basically you just have some space online, a bit of encryption, and some way to ensure that all clients of a user are synced, nonetheless building your own service can be a hassle as I have found out.

Currently I’m using SVN as a workaround. I just sometimes forget to commit, which leads to me having old data and no access to updated files. Instead of starting from scratch and build strange stuff with rsync I had a look if there were any open source solutions out there. And I was pleased to find a few. Unfortunately they are almost all in the “I have an idea and a few mockups” stage, maybe even some beta version that does something, but nothing really “production” level. Another downside was that most of them either supported Linux or Windows, but often not both.

One open source solution I stumbled across is iFolder, seasoned software supported by Novell, with clients for Windows, Linux and Mac. And if all else fails you can use the web interface to access your files. Admin and User web interface are both easy to use. I don’t remember how I stumbled across it, but I do know that it’s not one of the main products you see named when searching for alternatives to Dropbox.

If you have SuSE as an OS, then installing the software is immensely simplified due to SuSE belonging to Novell, just follow the SuSE instructions on the website. I prefer debian based installations and found Ubuntu instructions at help.ubuntu.com that were written mainly for Ubuntu, but should work with any debian deviate.

I’m not going to repeat anything from the installation instructions, they already did a fine job of documenting everything. One thing is important however: the Client 3.8.0.0 just plain doesn’t work. unfortunately it is the one linked from the website. Go to the sourceforge repository, and download the newer 3.8.03 client. That one works like a charm.