It is all about me!

you have nothign to prove, except theorems

So I ran Newman’s Gibbs sampler for LDA on a subset of 2000 documents from the arXiv and lo and behold already in such a small set we start to see useful topic descriptions …

general QM topics
[t1] operator function equation space phy form exp quantum relation hamiltonian representation solution …
[t2] set vector operator matrix theorem space element basis map unitary product group …

Cavity QED?
[t3] field atom cavity level frequency phy atomic fig rev ion laser transition …

Non locality
[t4] states entanglement phy entangled rev lett pure system bell local qubit mixed …

Quantum error correcting codes
[t5] qubit quantum error operation computation control gate spin number gates circuit computer …

Quantum optics
[t6] photon mode measurement phase beam rev fig phy single detector noise coherent …

Quantum information theory and crypto
[t7] quantum information bob alice algorithm classical protocol probability channel bit measurement problem …

other general….
[t8] wave cos sin potential spin function particle energy equation case electron momentum …
[t9] quantum theory measurement probability mechanic physical system space experiment classical particle physic …
[t10] system quantum evolution model hamiltonian phy dynamic initial classical decoherence density interaction …

very cool…

I love debian. Debian has been my friend for most of my adult life.
Sure I run XP on the laptop, but that is only because suspend didn’t work until 2.6.20 !
Apart from personal computing, debian is the only OS that I would trust to run on a server.

And all this time I have been thinking — how do I make money off of this knowledge?
I thought about becoming a linux support guy, which works and did that for a while.
It is good business, but it suffers from a serious fault in scalability.

You can only work SO much in one day.
As an employee the money making scales as X*h where h is the number of hours you work per pay period….
If you are good you can make X>30 maybe X>40. I don’t know? I made <20 since it was a “student job”, but I am sure people make a decent living from sysadmin / support jobs.
Whatever the X, even if you become a fancy shmancy consultant and get X=1000 … oh well then things are alright. ;)

Then you have the option of managing your own employees.
Then the money formula will scale with N, the number of employees you have.
(The contracts will scale but you will need more people so the two are linked)
I have little experience in this, but I bet it would be hard to find qualified people… linux isn’t that popular yet and you have to pay them well… so I dunno.
If someone has good management skills the IT middle management position in HIS OWN COMPANY could be a quality way of life.

Then we have my kind of ideas, where I have a server running and somehow whatever runs on that server is SOOOOO useful to my clients that they want to pay me money.
I just login once in the morning to look at the stats, check the log of bank transactions and then go have a day in the sun.
Machine works, Ivan chills.

So I have one of them ideas this evening. Please excuse the lengthy introduction.

Business model: start an ISP company that also sells freenx access to your own VM on our central server — with INIFINITE STORAGE and no transfer limits to and from your home computer
What will be the costs like:
power= 10$/month per server
storage= 100$/Tb
uplink internet= ?
bell lease= 22$/month per customer

why download/stream from home when you can stream to one of our media proxys and get faster speed
we can cache most content …
maybe even run a bittorrent tracker –> not to THROTTLE the users but to help them get more stuff

urLinuxVM.com
yourlinuxvm.com ?

It is almost finished now.
I am walking in the office (actually I have been here for several hours) and I’m going to get to business.
Just finishing the write ups: project report and the leterature review

What am-I talking about?
I am talking about the In-Search maga robotic browser something something…..

I started with a simple idea (2 weeks ago), but kept on adding features and doing quick upgrades here and there. In the end the project is done, but the write-up is far from completed. This is the last thing that remains to be done. I have good music, good coffee high and a good 2h of time ahead.

Do I have time to go to wing chun? Miss a class… but it is probably worth it so I will be done with this.
We will see…

Tomorrow around this time it will be all done.
Sure I may not have the crawler running yet, nor finished the review for the exam tomorrow but one thing is for sure…. it will be done sometime tomorrow. That is what is so great about finals… they are final :)

Ok. I am going to finish the literature review right now. Then go do some revision with Arthur and then come back home for some more coding… so I can e-mail the TA tonight a .tar.gz with some stuff in it.

Here is the absolute short list of options:

1. montreal crawler: gets results from google local and moves along the street
   what is that good for? : has more of a “robot” feel to it.
   what is that bad for? : presenting the data inevitably will have to involve a google maps interface which could make it more complicated
2. backlinks resumes: google search, find pages that link to top results and display a little summary for each page based on anchor text from these pages.
   what is that good for? : uses the fundamental idea of backlinks
   what is that bad for? :

It is time to pick one of these — because I clearly can’t do both.
Ok — i have decided — we are going for the backlinks project!

Let’s get coding.

So I have been adding some of the static pages to the CQIL website. It is a fun job with Django — editing some templates and pulling in the data from the db. Easy. Simple, but no simpler than it should be. Or is that so?

How can we make web-dev simpler?
Let’s figure out what we want first:

1. Rapid develeopment
2. No repetition of work
3. Using standards (easy to learn, portable knowledge)
4. Extensibility (framework should help you do the easy stuff, but get out of the way when you want to do custom stuff)
5. Editable through the web
6. Easy

I want to focus on the last point. In particular, given that there are several people involved in the construction of the website, each of these people’s tasks has to be as easy as possible. For the sake of simplicity I will use only three roles for my explanation:
Web-dev roles

• Programmer: (knows:python,php,ruby,mysql…) backend, frontend, db, everything you can think about. Dev, maintenance, optimization, deployment.
• Designer: (knows:photoshop, css, typography+ a little js) Is responsible for the site layout, usability, he/she must produce the css and the associated images.
• Content Writer: (knows: English) This person will produce all the content on the website. Articles, about-pages, news items, product descriptions.

Ex1: wordpress
The programmer is the wordpress team. The designer is the wordpress team or some other site where you can download wordpress themes. The content writer is you. It is a pretty sweet deal too since the whole stack is free :)
What makes wordpress work so well is that there is essentially one content type: the blog entry. So we have a very targeted CMS with rich features well targeted to help you write/maintain you blog. Great if you want to write a blog. Not so great if you want to write something else… like journal for example, or a gamebook.

Ex2: Plone+Zope
Here the programmers are the Plone+Zope team which have produced a lot of different content types like events, articles, news briefs, web pages, etc… If you want you can create your own content types (good luck! this might help).
The designer’s job… oooh boy… poor designer who will have to learn A LOT about the internals of Plone and write a ton of XML files to overwrite the default stylesheets. Also — and above all, I am questioning how efficiently he/she can do her job when all she is allowed to do is move around the servelets and portlets.
As for the content writer — hey he get’s a sweet deal with a very rich set of content types that will probably do 90% of the things he wants to do. He can edit things with point and click. Yeeey CMS done right!

Ex3: drupal or joomla or some other CMS
I don’t know enough about these to make an opinion but my guess is that they have similar problems to the Plone stack. Too much complication for the programmer/designer. This is of course not an issue if you spend a few months learning about the internals — surely then you will be a lean mean content type producing machine and will be able to handle lots of contracts from high paying clients :)
I should look into these one day …

Ex4: django
Django goes a LONG way towards the goal. In particular there is a split between the designer’s job (write templates) and the programmer’s job (write views). I can’t say I am a big fan of the django templating language — but I will say that it was very easy to learn and it had every construct I ever needed and nothing more. I guess ANY templating language that has good documentation and examples (like the django docs) is good enough for the task.

So while Django is nearly there there as far as the perfect web-dev platform requirements go, it is still not perfect. In particular these are the kinds of things I would like to have:

1. More intelligent blocks/inherit/includes in the templates
2. An interface between the content writer and the designer
3. An interface between programmer & designer (right now I put as the second line of my template comments like: {# REQS: person_list, page_title #}, which tell me which variables are required for this template.
4. Something that writes javascript for me :)
5. Widgets for each field type like Plone has
6. Through the web editing of everything like web.py

I am not saying these should be included in Django per-se. It would be perfectly fine if these are external projects. If I had time, I would work on this precisely, because the availability of a front-end framework for UI stuff can really make a difference for fast development.

So what DO I want specifically…
Here is what I envision as the “ideal scenario”.
1/ The programmer meets with the client and figures out what content types will be needed and how the site will work in general.

2/ The designer is given “building clocks” like div id=”homepage_text” and div id=”homepage_sidebar1″ with lorem impsum etc. instead of real content. He produces the layout working only with these blocks and changes the “sample” text to include examples of special functionality like ul’s that are actually the site navigational menu. etc…

3/ The content writer visits the page and edits-in-place the same building blocks using some simple markup language like ReST, Markdown or a wiki variety.

The essential thing I want is that all three parties can communicate using the same reference the “building blocks”. This would make the design super modular — I think and help with rapid development and code reuse.

For example, this is how I would like the homepage of the CQIL website to be rendered. First there is some general “top level” template base.html and the content-only file something like

/cqil/content/homepage.ReST

{% extends "two_column_layout.html" %}

{% block leftcolumn %}
{% filter restructuredtext %}
Welcome
=======
this is the homepage... alsjkalksj alks jaklj
{% endfilter %}
{% endblock %}

as you can see, there is an intermediary “connecting” template that produces the two column layout. This makes for VERY simple editing by the content writer if he/she had change the content.

Wow… I am pretty impressed. As I was writing this post I figured out that half of what I want is already possible with Django out of the box! Sweet.

What is missing then — well some javascript code to allow editing of the blocks though the web. This will be done with Jeditable next time I sit down to work on this. And I need to figure out where exactly I want the div-text to be stored. In the database? Sure that would be nice… but filesystem is also nice since you can edit with vim and or WebDAV. Can there be two versions — one on disk and one on db that mirror each other? If you change on disk it updates the db, if you change through the web it outputs a new version to the fs. Hmm… clearly some notion of version control is necessary like in wikis.

Ok… enough of this brainstorming. Good ideas… more code necessary ;)

it works surprisingly well…

moving away from the bash “one liners”
consider the following simple python script:

to backup your windows disk drive D:\ over ssh:
$ rdiff-backup –exclude-globbing-filelist doc_backup_exclude.txt /cygdrive/d/
ivan@192.168.2.133::/storage/master-laptop-backup/
(assuming you have cygwin with ssh & rdiff-backup)

doc_backup_exclude.txt
should contain things you don’t want to back up
one folder or file per line…
spaces in filenames need to be escaped
ex: /cygdrive/d/My\ Music/Some\ Folder\ With\ Spaces

henjoy…

Given that it is a rainy day today in Montreal, I decided to wake up early (like 10am :) and start organizing my papers. I have about 10 different avenues of academic research (QIT, neural networks, graph algorithms,etc…), about 10 business ideas and about 20 odd community project ideas. For each of these ideas there is some printed material, a bunch of shortcuts on all of my computers, brainstorming notes in notebooks and a bunch of scrap papers that I have kept “just in case” some crucial deep thought happens to be recorded there.

So I said *nuff of that* — let’s cleanup this mess and organize.

So of course, the first thing that I do is write a python script that will help me get it done :)
So now instead of organizing ideas, I am meta-organizing ideas….

But wait — there is more — as soon as I have this script working (which I must say I am fortunate enough was quite simple) I get an idea for a new Project. So of course — how best to describe your idea than with the project-admin.py? So now the python script is describing itself…

Here is the output: idea collector

Time to hit the paper stack organizing task before I go into another tangent….

single python script
that is an entire lesson

ex:

Welcome the UNIX and the command prompt in general.
It is a good place to be, with lots of things you can do.
Some people would say "efficiently"...

First though you have to get through the basics.
You must earn your yellow belt...

Press any key to continue ...


user@server$ | < --- blinking


First the basic existential questions:
whoami, list of things here, print working directory, etc...   

Press any key to continue ...


user@server$ |


type `whoami` into the window on the right


user@server$


BRANCH:
LISTEN TO: [whoami, pwd, list, ls, print working directory, window]
OTHERWISE ALLOW SHELL USE WITH ENV in CHROOT…

I just finished my Optimization assignment where I had to program the steepest descent and Newton methods of optimizing a function. I chose to do the coding in sage.

I can’t say that it was a good experience, but not because the software wasn’t good — rather there were so many cool things to try out and play with that I wasted a lot of time.

The interface is still a little rough around the edges. Take for example the matrix methods — they exist but you have to explicitly say over which field you are building your matrices….
m = matrix(RDF, [[1,2], [3,4] ])
where RDF stands for “real double float” — without that my matrix inverses were running into DivideByZero problems.

Also a vector v was not iterable — you couldn’t just do v[0], but that was easy to get out of with v.list().

Overall though — it is an amazing amalgamation of math software and a very promising platform. The notebook interface (web2.py + twisted server) is VERY powerful with latex integration + graphics + sharing and all.

Give it a few more years and this will be the leading math package out there!

PS: You can see the “published” notebooks here. If anyone wants to try it interactively then should email me and I can give them a demo account on the cube notebook.

I finally got it!

First the publications/models.py:

from django.db import models
from django.contrib import admin

from cqil.publications.manager import SOCSmanager

# This is an auto-generated Django model module.
#
# generated using:
# PYTHONPATH=".." ./manage.py inspectdb  --settings=cqil.pub_settings >> publications/models.py
#
# You'll have to do the following manually to clean this up:
#     * Rearrange models' order
#     * Make sure each model has one field with primary_key=True
# Feel free to rename the models, but don't rename db_table values or field names.
#
# Also note: You'll have to insert the output of 'django-admin.py sqlcustom [appname]'
# into your database.

class SOCSPub(models.Model):
    # custom manager
    objects = SOCSmanager()

    # introspected
    publication_id = models.IntegerField(primary_key=True)
    type = models.CharField(max_length=13, blank=True)
    author = models.TextField(blank=True)
    editor = models.CharField(max_length=100, blank=True)
    title = models.CharField(max_length=200, blank=True)
    booktitle = models.CharField(max_length=200, blank=True)
    pages = models.CharField(max_length=20, blank=True)
    chapter = models.CharField(max_length=100, blank=True)
    organization = models.CharField(max_length=100, blank=True)
    publisher = models.CharField(max_length=100, blank=True)
    address = models.CharField(max_length=100, blank=True)
    school = models.CharField(max_length=100, blank=True)
    institution = models.CharField(max_length=100, blank=True)
    number = models.CharField(max_length=20, blank=True)
    month = models.CharField(max_length=10, blank=True)
    year = models.CharField(max_length=10, blank=True) # This field type is a guess.
    note = models.TextField(blank=True)
    author_uid = models.CharField(max_length=20, blank=True)
    journal = models.CharField(max_length=100, blank=True)
    url = models.CharField(max_length=100, blank=True)
    ps = models.CharField(max_length=100, blank=True)
    pdf = models.CharField(max_length=100, blank=True)
    shown = models.CharField(max_length=5, blank=True)
    series = models.CharField(max_length=100, blank=True)
    inpress = models.CharField(max_length=5, blank=True)
    socstype = models.CharField(max_length=11, blank=True)
    volume = models.CharField(max_length=100, blank=True)
    class Meta:
        db_table = u'publications'

    def __unicode__(self):
        return self.author[:30] + ", " + self.title[:30] + ", " + unicode(self.year)

admin.site.register(SOCSPub)
~
~
~

Nothing interesting here — just a bunch of fields as extracted by the introspection mechanism.
(Note that during the introspection phase, I set the central settings file to point to the external db, not the local one.)

Now the key fun stuff happens in the manager. At first I read this article and this discussion, but I never managed to get it to work. Today I found this blog post by Eric Florenzano who really does things the right way. I adapted his code a little to keep it simple (since I only have one foreign db).

Here is the code of my publications/manager.py:

from django import db

# from eflorenzano...
from django.db.models import sql
from django.conf import settings
from django.db.transaction import savepoint_state
from  django.db.backends.mysql.base  import DatabaseWrapper
try:
    import thread
except ImportError:
    import dummy_thread as thread

# a dict with settings for the SOCS db
from cqil.publications.socsDBinfo import SOCSsettings

class SOCSmanager(db.models.Manager):

    # found my solution here:
    # http://www.eflorenzano.com/blog/post/easy-multi-database-support-django/

    # this function is used to "switch settings" for SOCSdb
    #
    def get_db_wrapper(self):
        database = SOCSsettings     # a dict of settings-like items
        backup = {}
        # save my settings
        for key, value in database.iteritems():
            backup[key] = getattr(settings, key)    # save setting
            setattr(settings, key, value)           # set to new one
        wrapper = DatabaseWrapper()
        wrapper._cursor(settings)
        # put back settings
        for key, value in backup.iteritems():
            setattr(settings, key, value)
        return wrapper

    def get_query_set(self):
        qs = super(SOCSmanager, self).get_query_set()
        qs.query.connection = self.get_db_wrapper()
        return qs

    def _insert(self, values, return_id=False, raw_values=False):
        query = sql.InsertQuery(self.model, self.get_db_wrapper())
        query.insert_values(values, raw_values)
        ret = query.execute_sql(return_id)
        query.connection._commit()
        thread_ident = thread.get_ident()
        if thread_ident in savepoint_state:
            del savepoint_state[thread_ident]
        return re
~
~
~

Last but not least here is the external db settings file publications/socsDBinfo.py:

# SPECIAL SETTINGS for SOCS publications remote DB 

# following the howto here:
# http://www.eflorenzano.com/blog/post/easy-multi-database-support-django/

SOCSsettings = dict(
    DATABASE_ENGINE     = 'mysql',              # 'postgresql_psycopg2', 'postgresql', 'mysql', 'sqlite3' or 'oracle'.
    DATABASE_NAME       = 'SOCS',               # Or path to database file if using sqlite3.
    DATABASE_USER       = 'theuser',             # Not used with sqlite3.
    DATABASE_PASSWORD   = 'thepass',             # Not used with sqlite3.
    DATABASE_HOST       = 'socs.server.com',   # Set to empty string for localhost. Not used with sqlite3.
    DATABASE_PORT       = '',                   # Set to empty string for default. Not used with sqlite3.
)
~
~
~

The key flaw in my original thinking was that I was trying to set the special “settings” as keyword arguments to the DatabaseWrapper object — since the underlying Python DB api usually does things this way. So I figured it must work… After consulting the django source code for _cursor I finally figured it out. Thanks Eric!