Site moved

This site has been moved to http://vincenzo-ampolo.net .

Categories: Uncategorized

Javascript/css loader

If you are a webdev you probably know that, sooner or later, you will need a way to load css and javascript asynchronously when you need them, and maybe track what you have loaded such that you don’t reload the same resources once again when you need them in other component.

ExtJs has that feature: the very well designed Ext.Loader can load arbitrary javascript files just looking on the configured paths of the different resources. On the other side of the most common libraries, Jquery has the $.getScript() function that does exactly the same thing, in a more dummy way. It just loads the javascript in the path passed as argument.

None of them are able to handle css, tho. So I wrote a small, library independent function to load a javascript or css dynamically. Here is the code:

var loadfile = (function(){
    //our hash table to keep record of already loaded files
    var loadedfiles = {};
    //TODO add callback
    return function(filenames, callback){

        if(!(filenames instanceof Array)){
            if(typeof(filenames) == 'string'){
                filenames = [filenames];
            }else{
                throw 'filenames is not an Array or a string';
            }
        }

        var map = {
            js: function(filename){
                var tag=document.createElement('script');
                tag.setAttribute("type","text/javascript");
                tag.setAttribute("src", filename);
                return tag;
            },
            css: function(filename){
                var tag=document.createElement("link");
                tag.setAttribute("rel", "stylesheet");
                tag.setAttribute("type", "text/css");
                tag.setAttribute("href", filename);
                return tag;
            }
        },i,length,type,tag,filename, ext;

        length = filenames.length;
        if(length>0){
            for(i=0; i<length; i++){
                filename = filenames[i];

                if(loadedfiles[filename])
                    //continue if we have already loaded this file
                    continue;

                type = filename.lastIndexOf('.');

                if(type === -1)
                    throw filename+' is not a valid filename with extension';

                ext = filename.slice(type+1);

                //create correct tag
                tag = map[ext](filename);

                //append to head
                document.getElementsByTagName("head")[0].appendChild(tag);

                //update hash table for files we've already loaded
                loadedfiles[filename] = true;

                //call callback
                if(callback && typeof(callback) === 'function')
                    callback(filename);
            }
        }
    };
})();

The loadfile function uses closure to store an hashmap of the loaded file and accepts an single string or an array of strings representing the paths to load. It also accepts an optional callback function that will be called if and when the script has been loaded into the page.

Feel free to use it as you wish. And if you like, give me some credit!

How to make usb disks readable by all users on Raspbmc

In a past article I wrote about a comparison of Openelec, Raspbmc and Xbian. That article hit a record in terms of number of readings and comments on this blog. I spoke about some modification I had to do to Raspbmc to make it usable for my needs but I din’t show them. In this article I’m going to fill this gap.

In order to make the usb drives be mounted with writing permissions to all the user of the system I’ve modified the /etc/udisk-glue.conf as following (in bold are the part that I’ve added:

filter disks {
 optical = false
 partition_table = false
 usage = filesystem
}

match disks {
 automount = true
 automount_options = { sync, noatime, <strong>"dmask=0", "fmask=0"</strong>}
 post_insertion_command = "udisks --set-spindown %device_file --spindown-timeout 1800 --mount %device_file --mount-options sync,noatime,<strong>dmask=0,fmask=0</strong>"
}

And I had to let udisk-glue run as root as following:


# udisks-glue
#

description "udisks-glue for udisks"

start on started dbus
stop on (runlevel [06] or stopped dbus)

expect fork
respawn

#exec su - pi -c "udisks-glue"
exec udisks-glue

(I’ve changed udisk-glue such that It’s run by root)

Since there are a lot of folks out there that are using this distro I think it’s is worth sharing this knowledge. If some dev is reading this, please consider to propagate this changes (or do equivalent ones) to the distro in order to increase flexibility.

Categories: hacks Tags: , , ,

Openelec vs Raspbmc vs Xbian

Update

In less than 12 hours after being public this article hit 1500 unique visitors. I received a lot of comments and thus an update is needed. You will find original part that has been corrected under stricke through text.

It’s now been a month that I’m playing with a borrowed (thanks to my flatmate Spencer Matteson) Raspberry Pi. Didn’t have time at the begininnig but as soon as I started connecting to a tv and sharing with all my flatmates it was fun from the first second.

I’ve been reached a stable configuration after X ( read as x big) number of researchs, hacks and tries. In this blog post I want to share my impressions and my researches about Raspberry Pi as a mediacenter with anyone that reads this blog post.

Raspberry Pi

It’s a cool toy, nothing to say about that. Even if people on the internet says that it lags in playing 1080p video in all my tests I didn’t find any lag. They pop up as soon as you start playing a 1080p video and doing netwoking intense operation like starting a rsync with a remote server or downloading 4/5 torrents.

If you have 25$ buy it. It’s worth the time and money you will spent on it. Due to the fact that it gets powered by a micro-usb connector (we are talking about 1A at 5V) don’t expect to connect a portable usb disk without power supply to the Raspberry. It will simply not going to work well and keeps rebooting just because power is too low to keep on the hard drive.

In my configuration I just connected to the Raspberry pi a ethernet cable, a HDMI cable and a usb hard drive with its own case and power supply.

Distro challenge

Openelec

Version tested: 3.0 RC2

Openelec was the first distribution I’ve tried. It’s easy, smooth to install and to use. It just works (TM). It has only one goal: be the best mediacenter distribution. And It just does it very good. With openelec you don’t have to worry about updates but as a con, you miss a package manager and everything of a linux distribution actually. It’s very stable and if you just need a mediacenter I will use this.

There is openssh installed but you will barely need it, I just used for a couple of reboots. The XBMC repository is already set and you can install any kind of software that is available directly from the UI. The problem is that you will not find anything advanced like a webserver or any other package you will find in other distributions. The farest I went while using it was to install transmission bittorrent client directly from XBMC repositories from the UI. It went well and worked pretty smooth. But transmission just uses http and does not support https so exposing it outside your personal network is not wise. And since you don’t have a webserver you can’t actually put an apache configured as https trasparent proxy in front of transmission.

Also I was really disappointed when I figured out that It doesn’t support changing ssh root password as its FAQ states. Since I soon wanted to do more with Raspberry pi that a simple mediacenter this was clearly an issue. I than tried Raspbmc.

Raspbmc

Version tested: 1.0 final installed on 3th of Feb 2013

Seriously. Don’t do that. Don’t spend your time in trying to figure out to make this distro work. It’s bloated. As soon as I installed I liked the really nice self installer and the cool splash screen while the system is loading. But apart from that, underneath you have a broken apt and package repository.

I tried to install the *byobu* package. When it was installing it’s dependence, *screen* there was a detected /deadlock in the init system/ so it refuses to install it. What?! (as a comment demonstrated this issue seems to have been fixed after this post. I’ll try myself as soon as I’m nearby my Raspberry). After that I installed apache2 and transmission and linked the two together making apache2 being a transparent ssl proxy. It didn’t work. It appears that Raspbmc mounts the drives as the pi user which is the one that runs the xbmc application using writing permission only for the pi user. The problem is that transmission runs as debian-transmission user. So it is unable to write on my external usb disk! After tweaking with /etc/udisk-glue.conf configuration I managed to mount the devices as root and assigning a write permission to all the users.

(The best part was when I tried to install some software from XBMC repository. I couldn’t because the repo was empty. Bart Otten, webmaster of Xbian.org and Superrepo.org contacted me privately saying that this issue is related to the XBMC repo. It can happen in any distribution that has XBMC. So if you don’t find software in your XBMC addon catalogue, just wait until the repository is up again.)

It may be the fact that Raspbmc is a one person project but the fact is that I found it really unstable and not flexible. If I were Sam Narzako (the man behind Raspbmc) I would open the project to more people or, better, contribute to the Xbian project.

Sam Nazarko politely commented that Raspbmc is not a one man project. I apologize for the misunderstanding but I got this information from the raspbmc.com about page that states that “Raspbmc is created and maintained by Sam Nazarko, a 19 years old student from London”.

Xbian

Version tested: 1.0 Alpha 4

Xbian was really easy to install. After downloading its image and put in a sd card as its website states first time that it loaded on the Raspberry pi to stretch the partition in order to use all the space in the sd card. The system was ready in less than five minutes and I was able to install byobu, apache2 and transmission without problems. Also the usb hard drive is automounted allowing anyone to create or read any file (chmod 777).

I then installed the XBMC watchdog plugin in order to keep the library up to date with new additions. The system resulted to be really stable.

I found Xbian to be the most mature mediacenter distribution for Raspberry pi. It is easy to install and maintain but it preserve the flexibility of a debian based distribution. The only annoying thing is a menu that appears to the user anytime he/she logs in with ssh. I don’t think that this behaviour is wanted by an user that is connected through ssh because it tries to simply management. But who connects through ssh is not a newbie and thus does not need any simplification. Instead this menu alter the expected behaviour that the user imagine about connecting through ssh.

Configuring apache2 as transparent proxy for transmission

In order to simplify configuration, the configuration for apache2 as transparent proxy for transmission is the following:

<VirtualHost *:443>
 <proxy>
    Order deny,allow
    Allow from all
  </proxy>

  SSLEngine On
  SSLCertificateFile    /etc/ssl/certs/ssl-cert-snakeoil.pem
  SSLCertificateKeyFile /etc/ssl/private/ssl-cert-snakeoil.key

  ProxyPass        /  http://127.0.0.1:9091
  ProxyPassReverse /  http://127.0.0.1:9091

</VirtualHost>

I’ve my new shiny mediacenter, how do I control it from my sofa?

You can have a Raspberry pi with all the software you want, but you will never have a mediacenter until you can control laying on your sofa. For xbmc, which is the base software in all the distribution I just spoke about, the remote control software is Yatse or Xbmc remote.

I tried both and I prefer Yatse. It’s much more complete and the UI is much more effective than the one in Xbmc remote. Also for functionality yatse seems much more powerful even if I didn’t analyze each specific functionality between the two remotes. You can install Yatse directly from goole play as like Xbmc remote.

They both work accessing your Raspberry pi via the network. All of the distribution I’ve shown have remote control enabled by default although both Openelec and Raspbmc use port 80 while Xbian uses port 8080.

I highly suggest to make Xbmc scan through your media files in order to populate your library. In fact library is easy accessible, sortable and playable from yatse and It will save a loot of file in finding the right file expecially when it’s in a directory polluted by other media files.

Alex, from a comment, says that XBMC Remote for iOS works very well, I didn’t test it personally. YYou can find it on the app store.

How to run OpenVpn and a webserver (Apache, Nginx, Cherokee, etc) in the same port

http://www.hotforsecurity.com/wp-content/uploads/2012/10/The-New-York-Times-Website-Blocked-in-China-1.jpg

If you have ever used OpenVpn you know that a lot of filtered network do not allow to connect to other ports rather than 80 and 443 TCP. If you have your OpenVpn server in the same machine alongside with a Webserver this may be a problem because you want to use port 80 and 443 for your webserver and also the openvpn.

OpenVpn developers are smart people and they knew that this situation is not so odd so they invented a special option to deal with it. Thanks the article at raniersblog.org I found the port-share option that just redirect all the packets that are not vpn ones to another host:port pair. This allows to run your webserver in a different port, let’s say 8443 and make OpenVpn run on the 443 TCP. If packets are not for the vpn they will get redirected to the 8443.

To do so just configure your webserver to listen to port 8443 and add this configuration to the openvpn configuration file (usually /etc/openvpn/server.conf):

proto tcp
port 443
port-share 127.0.0.1 8443

Then restart openvpn and your webserver and you should be all set.

Categories: HowTo, sysadmin Tags: , , , ,

Python Bunch object or How to make dictionaries act like objects

Python dictionaries are really powerful, but sometimes you need an object that is a dictionary but it’s also an object which keys are methods of the same object.
Now think if you can have a class called Bunch that makes something like this possible:

bunch = Bunch(a = 7, b = 3, c = 'hello')
In [5]: bunch.a
Out[5]: 7
In [6]: bunch.b
Out[6]: 3
In [7]: bunch.c
Out[7]: 'hello'

But it’s also more powerful than this:

In [9]: d = {'a.b.c':1, 'a.c':2}
In [10]: bunch = Bunch(**d)
In [11]: bunch.a
bunch.a      bunch.a.b.c  bunch.a.c
In [13]: print bunch.a.b.c
1
In [14]: print bunch.a.c
2
In [15]: print bunch['a.b.c']
1

You can also use the dot notation to describe an hierarchy of objects and the Bunch object will take care of it

It’s not a dream, I’ve just implemented this python Bunch Object and this is the code:

class C(object): pass

def rec_getattr(obj, attr):
    """
    Get object's attribute. May use dot notation.
    """
    if '.' not in attr:
        return getattr(obj, attr)
    else:
        L = attr.split('.')
        return rec_getattr(getattr(obj, L[0]), '.'.join(L[1:]))

def rec_setattr(obj, attr, value):
    """
    Set object's attribute. May use dot notation.
    """
    if '.' not in attr:
        setattr(obj, attr, value)
    else:
        L = attr.split('.')
        if not hasattr(obj, L[0]):
            setattr(obj, L[0], C())
        rec_setattr(getattr(obj, L[0]), '.'.join(L[1:]), value)

class Bunch(dict):
    def __init__(self,**kw):
        dict.__init__(self,kw)
        self.__dict__.update(kw)
        for k,v in kw.iteritems():
            rec_setattr(self, k, v) 

Comments, suggestions, patches are welcome :)

download list of files from a remote ssh/sftp server

I’ve approached this problem before. I’m on a server and I want to copy some directories from another server using scp. You have to write down all the paths of the directory you want of the first server, then go to the second, then do a command like:

scp -r server1:"directory1 directory2 directory3 ..." .

Which is easy with simple filenames but as soon as you need directories in different paths or long named directories it will be pain. And you don’t have shell completion. You could say: “Set up key authentication and you will have shell completion on the remote server”. Well, I don’t wanna set up keys for a server and I want a more practical way to solve this.
I wanted to connect through ssh from nautilus (just type ssh://server on the path bar), surf my remote directories, then download what i need to a second server. Well I did a python program to do that.
It’s pretty straightforward. You connect with nautilus to a server, select all the directories you want then copy them. Open the shell of another server, then do: python3 download.py -s server1 CTRL+V
The CTRL+V will copy all the filename absolute paths straight to the shell. Just hit enter, it will ask for password, just give it and you are done. It will start downloading all the list of directories of files you want from server1
This is the python program that makes this possible.

<pre>
#!/usr/bin/python
# -*- coding: UTF-8 -*-
'''
This programs allows easy copies of remote directories to the local path using scp

        usage:
                python3 download.py [options] list-of-files

        options:
                -s, --server server:set the server instead of deducing it

@author: Vincenzo Ampolo <vincenzo.ampolo@gmail.com>
'''

import sys
import os
import getopt

import urllib.parse

def shellquote(s):
        """escapes the given string s such that it can be used safely in a shell
        """
        return "'" + s.replace("'", "'\\''") + "'"

def main(argv=None):
        if argv is None:
           argv = sys.argv

        opts, extraparams = getopt.gnu_getopt(argv[1:], "hvs:", ["help", "server"])

        verbose = False
        server = None

        for o, a in opts:
                if o == "-v":
                        verbose = True
                elif o in ("-h", "--help"):
                        print(__doc__)
                        sys.exit(0)
                elif o in ("-s", '--server'):
                        server = a
                else:
                        assert False, "UnhandledOption"

        l = []
        for path in extraparams:
                url = urllib.parse.urlparse(path)
                if not server:
                        server = url.host
                quoted_path = shellquote(urllib.parse.unquote(url.path))
                l.append(quoted_path)
        files_to_download = ' '.join(l)
        cmd = 'scp -r %s:"%s" .'%(server, files_to_download)
        os.system(cmd)
        return 0

if __name__ == '__main__':
    sys.exit(main())
</pre>
Categories: Programming Tags: , , , ,
Follow

Get every new post delivered to your Inbox.

Join 540 other followers

%d bloggers like this: