Jul 04

Share Storage across multiple Amazone EC2 Instances

Today I came across a very huge problem with AWS and their very cool cloud solutions. Currently my work is to setup a live environment in my company, which is scalable and which decreases the risk of failure and non-availability. Using the Amazone Web Services is very straightforward in that case, so I’ve made a plan ofhow to setup the servers to enable such a system using amazones services RDS, EC2, ELB, EBS and S3:

Everything seems to be very clear at this point. We have a master database which replicates to some slaves. These slaves are dedicated to the EC2 Instances. A final Load Balancer will manage the access across all EC2-Instances.

The problem came up when you imagine the following situation: You have installed for example a wordpress, which is hosted in the cloud using the plan above. Now a user uploads an image into that WordPress. Since the EC2-Instances are all launched from a AMI, which was created at the start, all EC2-Instances have the same configuration, but not the same EBS Volumes where your WordPress is saved to.

The Solution to this came up to me after a short time: Network File System. With NFS, we can create a shared storage, which is used by every EC2 Instance. The plan was to use the first EC2 Instance as a NFS-Server. Every new EC-Instance acts as an NFS-Client, which only means that the NFS-Client mounts the Shared storage at startup.

Server Livesystem using Amazone Web Services

Now I want to give you a very very basic How-To for this particular setup without going into depth in things like security. All steps are tested with Ubuntu 12.04 LTS 64 Bit:

Step 1 – Setup NFS-Server

Install the package nfs-kernel-server on the Server EC2 Instance

sudo apt-get install nfs-kernel-server

Open TCP and UPD Ports in AWS-Console

TCP: 111, 2049; UPD: 111, 32806

Add a directory you want to share to the file /etc/exports, for example /mnt (here in read-only mode and in async mode to enable async access) – NOTE THAT NOW THE HOLE WORLD HAS AT LEAST READ ACCESS TO THAT DIRECTORY. USE A SPECIFIC HOST OR A COMMA SEPARATED LIST FOR THE * TO RESTRICT THE ACCESS!:

/mnt          *(ro,async)
Now restart the NFS-Service
sudo /etc/init.d/nfs-kernel-server restart
sudo exportfs
Step 2 – Setup NFS-Client
Install the package nfs-common on the Client EC2 Instance

sudo apt-get install nfs-common

Open TCP and UPD Ports in AWS-Console if this isn’t already happened

TCP: 111, 2049; UPD: 111, 32806

In order to mount the directory /mnt from the server into the own /mnt directory you have to run the following command:

sudo mount -t nfs ec2-ip.zone.compute.amazonaws.com:/mnt /mnt
If you want to unmount that volume for some reason:
sudo umount -f /mnt
Now simply add the mount command to the file /etc/rc.locale to make the mount process happen at startup of the EC2 Instance and you’re done.

Apr 03

Update package on Ubuntu using apt-get and apt-show-versions

Sometimes you have to update some installed packages to a specific version. In order to simplify this process, I’ll show you one solution.

First you have to install apt-show-versions with the following command:
apt-get install apt-show-versions

Once this is done, you can show all available versions for a specific package with the following very helpy command:
apt-show-versions -a -p [package-name]

I’ve done that on my server fpr the package apache2, which produces the following output:
root@*******:~# apt-show-versions -a -p apache2
apache2 2.2.14-5ubuntu8.9 install ok installed
apache2 2.2.14-5ubuntu8 lucid ftp.stratoserver.net
apache2 2.2.14-5ubuntu8.8 lucid-security ftp.stratoserver.net
apache2 2.2.14-5ubuntu8.9 newer than version in archive

In order to install a specific version, run the following command:
apt-get install [package-name]=[version]

In the apache2 example, in order to install version 2.2.14-5ubuntu8, I have to run the following command:
apt-get install apache2=2.2.14-5ubuntu8

These simple and easy steps should help you alot in your everyday life as a system administrator.

Mar 30

Zend Framework: Protect specific URL with password using .htaccess

Sometimes when programing in php using the famous Zend Framework, you have to provide a backend to your customer. When the backend does not need to have a real user management, but should be protected from public users (like always for backends ;) ), then this solution is an easy way to provide a simple login dialog:

Create an exact copy of your index.php file in the public folder of your Zend project named backend.php for example. Then modify your .htaccess file as following (This sample assumes that you have a module named backend in Zend so that you can reach your backends IndexController with the following URL: http://your.domain.de/backend/index/index):
RewriteEngine On
RewriteCond %{REQUEST_FILENAME} -s [OR]
RewriteCond %{REQUEST_FILENAME} -l [OR]
RewriteCond %{REQUEST_FILENAME} -d
RewriteRule ^.*$ - [NC,L]
RewriteRule ^.*(backend).*$ backend.php [NC,L]
RewriteRule ^.*$ index.php [NC,L]

<filesMatch "backend.php">
AuthUserFile /path/to/your/file/.htpasswd
AuthName "Enter Password"
AuthType Basic
require valid-user

Now you can create the .htpasswd file with the following command:
htpasswd -c .htpasswd admin

When you then visit your backend, you’re prompted for the password, but not when you access the public area.

Mar 22

Center a static div container using HTML and CSS

Centering a div container may be a complicated issue in the first look. However the solution is as simple as logic: With just a minimal amount of thinking, you can simply use addition and the position of the div tag to solve the problem.

Lets say that you have a simple page like this (I have to use these wrong square brackets :) , sorry for that):
[div id="to_center"]
I want to be centered...

In order to center the given div tag, you have to define a little small handy css block:
#xyz_main {
width: 500px;
position: absolute;
left: 50%;
margin-left: -250px;

As you can see we define a width of 500 pixels, then we use left: 50% to move the hole div in the middle of the screen (not centered yes, but starting in the exact middle of the screen).
After that we move it half of it’s defined width (250px) back to the left.

It’s not very comlicated in the end, but sometimes it’s not immediately clear.

I hope this helps somebody.

Mar 06

Export Meshes from Blender FBX to Unity 3D

Using Blender for Modeling 3D-Content is very interesting especially for indie game developer teams. Since blender supports an export to FXB, it’s also very interesting to use in combination with Unity 3D.
We’ve experiented a lot to get our models from Blender (v2.62 r44136) out into Unity 3D and we’ve seen many pitfalls when exporting models.

1. Casestudy: The Unitcube
It starts to get in trouble with the simplest test you can make: Create a new Blender Scene with the unit cube inside and simply export it to fbx with all settings left as they are:

Drop it into your Assets folder of your Unity-Project folder and look into unity – thats what you normally would see:

As you can see, the normals are not exported the right way when using the default settings. You can now play with the settings, but believe me when I say I have tested neary every combination, neighter works as expected.

2. Find the reason
Later I took a look into the python export script an then I saw whats the reason. Look at the next picture:

This image shows all normal vectors in blender, used by the unit cube. If you select one vertex you can see that 3 faces share this vertex. In fact the fxb export script does only export the darker blue normals and not the face normals. In other words, the exported fxb file does not contain 3 individual vertices with the 3 different facenormals, only the vertex normals are exported and when importing it into unity, the importer will create individual normals. In this step, the needed face normals are not present, which results in the given wrong visualization.

3. Use Blender to fix this, not Unity!
One way to get all meshes imported into unity is to not import normals in Unity, but to let them automatically calculate while importing. This in most cases will work like expected and will create sufficient results, but there are cases, where your created normals are not reproducable while importing.

To get rid of this using Blender, you have to use a special modifier, the Edge Split Modifier:

This modifier will make individual vertices for every face. BUT: You only have to add this modifier to the objects you want to export, you do not have to really apply it! This is done while exporting the meshes so you can model in blender as normal! You just have to make sure, that the edge split modifier is the last one in the list of modifiers added to the mesh.

Here is a screenshot of the unit cube normals with the edge split modifier:

Now each Vertex has it’s own normal (the darker blue one). For the Screenshot i had to really apply the modifier. Remember that you do not need to apply it, only add it. Now when exporting to fbx, make sure that the small box ‘Apply Modifiers’ is checked (default):

When everything is fine in Unity 3D, you should now see a perfect unitcube with the right normals and without calculating them using Unity 3D:

Finally when you know how to deal with blenders modifiers, it may become a very interesting tool for you. One thing to mention about: All steps provided here do not cause an increase of vertices, normals or faces in Unity!

I hope this will help some people out there to get started with Blender and Unity 3D.

Feb 02

Techdemo – The follower project of Hybrisma

The new Project Techdemo (name is choosen in a few days :) ) will replace the Project Hybrisma. The project Hybrisma was a very good idea, but it was too complex to develop and to manage as we’re only free people doing it. So we’ve developed a Techdemo concept for an MMORPG, which is small enough to develop and to manage. Up to now we’re 7 active people.

The Level we’ve planned is abount 400 meters x 400 meters in it’s size and we have a small story around it, which is extendable in many places.

The technical overview is simple:

  • The C++-TCP-Socketserver
  • MySQL for data storage
  • Unity3D is used for the Client
  • C# is used for the game logic in Unity
  • We have a central Wiki for the documentation of the project
  • We have a ticketsystem to manage our team and manpower
  • We have an administration backend written in HTML/PHP

The peoples we have are grouped:

  • 3 peoples in 2D-Graphics
  • 1 people (me) in programing (server and client)
  • 2 people in 3D-Modeling
  • 1 people in writing and Game-Design

If you’re interested in this project and if you can speak german (not needed if you are interested in programing :) ), feel free to leave a coment here or to email me at andy@hybrisma.de.

Feb 02

TCP-Socketserver: Client-Server Communication – The Development goes on

In the last month the TCP-Socketserver developed for the MMORPG idea has changed so much due to another project I’ve done. This other project was the test for the Server in case of performance and stability. Now I can say, I have a very very well and performant TCP-Socketserver, which can handle up to 50k Clients without doing broadcasting operatione between them, but which can call several routines like creating a new user and so on. Also the Server is connected to a Database (MySQL) for data exchange.

To reach the goal of 50k users you cannot simply go what you want. You have to carefully look at every single line of code to ensure that it is maximum performant and does not introduce additional work. My Server uses nonblocking sockets, which simply means that the read() or recv() functions immediately return the result and do not block until data arrives at the requested socket. Then for every client the Server caches the incoming data in a separat buffer. Every time when new data arrives for a connected socket, the server will try to detect whether a specific message is completely readable at the current state and with the amount of data available. If it’s not the case the server will collect more data until this goal is reached. When a full message is available/readable, it will be processed.

To maximize performance, I’ve developed a bitencoded protocol using only the amount of data needed to deliver the data between server and client. If you think about orientation updates of a character walking in a Level, it’s not that difficult to see that this can grow to a very large amount of data. Some client moves in the World. Lets say there are 100 clients walking around in one level. Every Character sends orientation updates to the server about 10 times per second (hopefully this amount is less than 10 in the final version with a more complex interpolation method in the client to remove flickering while direction and speed changes). Every data package with orientation updates is about 24 bytes long. When this package is send to the Server, the server itself will broadcast it to all the other 99 connected clients, which will give you 99*24*10=23760 Bytes of data, which is send over the network of the server. Always keep in mind that this is only the orientation update. Also you can calculate how important it is to reduce the amount of data of this package to the minimum. Also you can try to minimize the amount of clients, which need to know the orientations of other clients due to view frustum clipping, which is currently ignored in my version.

If somebody out there is interested about some additional informations, feel free to email your question to andy@hybrisma.de or leave a comment here.

Older posts «