Introduction

ShortMe is a url shortening service written in Golang.
It is with high performance and scalable.
ShortMe is ready to be used in production. Have fun with it. :)

Features

f**kstupid/version/healthhttp://127.0.0.1:3030/version

Implementation

Currently, afaik, there are three ways to implement short url service.

>>> import random
>>> import string
>>> random.sample('abc', 2)
['c', 'a']
>>> random.sample('abc', 2)
['a', 'b']
>>> random.sample('abc', 2)
['c', 'b']
>>> random.sample('abc', 2)
['a', 'b']
>>> random.sample('abc', 2)
['b', 'c']
>>> random.sample('abc', 2)
['b', 'c']
>>> random.sample('abc', 2)
['c', 'a']
>>>

Api

    *   Trying 127.0.0.1...
    * Connected to 127.0.0.1 (127.0.0.1) port 3030 (#0)
    > GET /3 HTTP/1.1
    > Host: 127.0.0.1:3030
    > User-Agent: curl/7.43.0
    > Accept: */*
    >
    < HTTP/1.1 307 Temporary Redirect
    < Location: http://www.google.com
    < Date: Fri, 15 Apr 2016 07:25:24 GMT
    < Content-Length: 0
    < Content-Type: text/plain; charset=utf-8
    <
    * Connection #0 to host 127.0.0.1 left intact

Web

The web interface mainly used to make url shorting service more intuitively.

For short option, the shorted url, shorted url qr code and the corresponding long page is shown.

For expand option, the expanded url, expanded url qr code and the corresponding expanded page is shown.

Superstratum Projects

short-me

Install

Dependency

  • Golang
  • Mysql

Compile

Database Schema

We use two databases. Import the two schemas.

  • shortme
    • Store short url info
  • sequence
    • sequence generator

Configuration

[http]
# Listen address
listen = "0.0.0.0:3030"

[sequence_db]
# Mysql sequence generator DSN
dsn = "sequence:sequence@tcp(127.0.0.1:3306)/sequence"

# Mysql connection pool max idle connection
max_idle_conns = 4

# Mysql connection pool max open connection
max_open_conns = 4

[short_db]
# Mysql short service read db DSN
read_dsn = "shortme_w:shortme_w@tcp(127.0.0.1:3306)/shortme"

# Mysql short service write db DSN
write_dsn = "shortme_r:shortme_r@tcp(127.0.0.1:3306)/shortme"

# Mysql connection pool max idle connection
max_idle_conns = 8

# Mysql connection pool max open connection
max_open_conns = 8

[common]
# short urls that will be filtered to use
black_short_urls = ["version","health","short","expand","css","js","fuck","stupid"]

# Base string used to generate short url
base_string = "Ds3K9ZNvWmHcakr1oPnxh4qpMEzAye8wX5IdJ2LFujUgtC07lOTb6GYBQViSfR"

# Short url service domain name. This is used to filter short url loop.
domain_name = "short.me:3030"

# Short url service schema: http or https.
schema = "http"

Capacity

unsigned bigint18446744073709551616LastInsertIdint64uint649223372036854775808
100,000,0002 ** 63 / 100000000 / 365 = 252695124

Short URL Length

2 ** 63

Grant

After setting up the databases and before running shortme, make sure that the corresponding user and password has been granted. After logging in mysql console, run following sql statement:

grant insert, delete on sequence.* to 'sequence'@'%' identified by 'sequence'grant insert on shortme.* to 'shortme_w'@'%' identified by 'shortme_w'grant select on shortme.* to 'shortme_r'@'%' identified by 'shortme_r'

Run

static./shortme -c config.conf

Deploy

Sequence Database

In the Flickr blog, Flickr suggests that we can use two databases with one for even sequence and the other one for odd sequence. This will make sequence generator being more available in case one database is down and will also spread the load about generate sequence. After splitting sequence db from one to more, we can use HaProxy as a reverse proxy and thus more sequence databases can be used as one. As for load balance algorithm, i think round robin is good enough for this situation.

In two databases situation, we should add the following configuration to each database configuration file.

  • First database
auto_increment_offset 1
auto_increment_increment 2
  • Second databse
auto_increment_offset 2
auto_increment_increment 2

replace into sequence(stub) values("sequence")

In cases we use three databases as sequence counter generator, we should insert a record for each table in two databases.

  • First database
auto_increment_offset 1
auto_increment_increment 3
  • Second database
auto_increment_offset 2
auto_increment_increment 3
  • Third database
auto_increment_offset 3
auto_increment_increment 3

replace into sequence(stub) values("sequence")
N
for i := range N {
    add "auto_increment_offset i" to config file
    add "auto_increment_increment N" to config file
}

So, sequence generator can be horizontally scalable.

Shard

With short urls increasing, many records are stored in one table. This is not an optimal mysql practice. In this case we can simply shard table to bypass this problem.

For example, we can shard according to the base integer using modula hash algorithm. This has a good distribution between tables. We can use 100 short tables with names like short_00/short_01/short_02/.. ./short_99. we can use pseudo code blow to determine which is the table to store the short url record.

baseInteger := sequence.NextSequence()
tableName := fmt.Sprintf("short_%s", baseInteger % 100)

There are many table sharding algorithms, we can shard table according to range id, user name and so on. If we use user name as the criteria to shard table, we can do some aggregate algorithm like how many records a user has created easily. This may also has some drawbacks such as if user Lily and user Lucy are sharded to different tables and Lily shorts about 1k urls Lucy shorts about 1M urls, then we may encounter the unbalance hash problem, i.e., some tables contains more records than others.

In conclusion, there are many factors to consider before we can make a decision which hash algorithm to use.

Statistics

Sometimes we may want to make some statistics about hit number, UA(User Agent), original IP and so on.

awkELK

Problems

  • long url may make the generated qr code unreadable. I have test this in my self phone. This remains to be tested more meticulous.
  • One demand about customizing the short url can not be done easily currently in shortme according to the id generation logic. Let's make it happen. :)

Orange Juice

I like drinking orange juice. I will appreciate that if you can give me a cup of orange juice. :)