Hello python

I’m a Java guy and I just can’t convince myself to learn a new language – because there isn’t a better one. Let’s look at python for a second, there is visually no structure. Why is having {} blocks a problem?

[python]
#!/usr/bin/python3

# Imports
import random

s1 = "Let’s predict the next mark six!"
print(s1)

genEntries = 4
sysRandom = random.SystemRandom()

while genEntries > 0:
genNumbers = 17
selection = []
result = []
# first fill selection array with 17 random numbers
while genNumbers > 0:
temp = sysRandom.randint(1, 49)
if temp not in selection:
selection.append(temp)
else:
continue
genNumbers -= 1
# then pick 6 numbers from selection
result = sysRandom.sample(selection, 6)
result.sort()
print(result)
genEntries -= 1

"""
Let me know if you win with this script!
"""
[/python]

Moving to EC2

Powered by AWS Cloud Computing

I had to give it a try. This blog is now moved to AWS. There are a lot of cool features but they’re sort of expensive. For example, if I put the database on RDS and memcache on ElasticCache, it will cost me $140/mo. Come on, running yum -y install memcached is not worth $50 a month!

Actually, all of this blog can fit into a single instance and I can enjoy the free usage tier, hopefully.

Being a cli-head, I gotta have some tool that doesn’t require a dozen mouse clicks. awscli is exactly what I need. To install it, provided you already have python 2.6 and pip installed:

pip install awscli

Then configure a profile and store the IAM access credential

aws –profile personal configure

Once the IAM credential is configured, I can proceed to doing a lot of things. Here are a few examples to get you started:

aws help
aws –profile personal s3 ls
aws –profile personal ec2 describe-instances
aws –profile personal ec2 describe-snapshots –owner-ids self

From within your instances, one can also query EC2 parameters using the ec2-metadata tool.

wget http://s3.amazonaws.com/ec2metadata/ec2-metadata

ec2-metadata -i -a
instance-id: i-4c24f511
ami-id: ami-baddecff

 

 

Updating RHEL to specific minor release

There are times when updating to the latest minor release is not preferred, maybe a software compatibility issue or just plain stupidity. On RHEL 6.1 and beyond, one can utilzie the releasever parameter to lock the update to specific one. For example:

yum –releasever=6.2 update

But what about RHEL5.x? Redhat suggested mounting the corresponding ISO and set it up as a local repository. Lots of work especially when it’s required for no good reason.

Here there seems to be another way to do it. First, using the update-to option to install the redhat-release package you need. For instance, I want to upgrade from 5.7 to 5.8:

yum update-to redhat-release-5Server-5.8.0.3

Thereis a little trick here, the last release digit needs to be guessed. So I tried from 5.8.0.0 and finally found 5.8.0.3 is available. It can be done with the yum info command.

Next thing is to prevent further update of the redhat-release package. Edit /etc/yum.conf and add

exclude=redhat-release*

Then run yum update as usual.

[[email protected] ~]# lsb_release  -r
Release:    5.8

References:

  • https://access.redhat.com/site/solutions/92383

s3cmd failed to upload

Using s3cmd 1.0.1, I was unable to upload a 2.7G file. It’s not even 5GB.

WARNING: Upload failed: /private.raw.tar.gz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=1.25)
WARNING: Waiting 15 sec…

Turns out there is a new feature with s3cmd 1.5 which is to split large files into multiple smaller ones, each one being 15M. Only then was I able to upload my file:

$ s3cmd --no-encrypt put private.raw.tar.gz s3://xxxxx
WARNING: Module python-magic is not available. Guessing MIME types based on file extensions.
WARNING: Redirected to: xxxxx.s3-us-west-2.amazonaws.com
private.raw.tar.gz -> s3://xxxxx/private.raw.tar.gz [part 1 of 188, 15MB]
15728640 of 15728640 100% in 26s 575.98 kB/s done

And it looks like the new version needs python-magic.

Got problem?

We deal with problems every single day. I’d like to think that there is a systematic way to solve them so we don’t make a bigger mess or we don’t let it become a routine job. We want to be able to say with confidence, that what doesn’t kill you makes you stronger. I try to follow 7 steps whenever possible.

problem-solving

Gather data: Don’t get ahead of yourself and try to speculate or fix anything. Use your time to gather as much data about the situation as possible. Don’t assume you know it. Measure twice and cut once. Some people are able to do this by instinct and building logics around that but being quick is not the objective here.

Control situation: Control the situation so that it does not get worse. Especially when it’s a critical situation, you want to hold it in place and buy time for the next steps.

Problem isolation: With sufficient information and a controlled situation, now it’s time to study the problem, break it down, and relate all the pieces. This is the only way to fix it accurately. Find the root cause.

Identify risks: When trying to fix a problem, it’s important to know if it will introduce new problems which make the original problem harder to fix.

Mitigate risks: Do a backup, setup safety net, give a heads up to others, get some sleep. Be equipped for what’s coming.

Prevent recurrence: Implement permanent fix in direct response to the root cause whenever possible. The idea is to fix it so that it never comes back in its original form. This is when you need to think – screw me twice, shame on me.

Learn from it: Review the incident. There is always things that can be done better or mistakes that can be avoided. If you walk away the minute the problem is solved, you learned nothing.

Partitioning MySQL

This is purely for the sake of partitioning.

In this example, I’ll quickly partition the wordpress comments table into partitions. The hash partition ensures data are evenly distributed across partitions.

mysql> alter table wp_comments partition by hash(comment_ID) partitions 3;
Query OK, 13 rows affected (0.04 sec)
Records: 13  Duplicates: 0  Warnings: 0

Now check that queries are only reaching out to necessary partitions

mysql> explain partitions select * from wp_comments where comment_id = 1\G
*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        table: wp_comments
   partitions: p1
         type: const
possible_keys: PRIMARY
          key: PRIMARY
      key_len: 8
          ref: const
         rows: 1
        Extra:
1 row in set (0.00 sec)

The hash function can be overridden with any function that returns an integer. For example, I can use mod(some_id,2) to divide some_id into two groups – even and odd.